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Communication plays an 
essential role in disease 
prevention and health 
promotion. Programs designed 
to promote changes in health 
behaviors and to encourage 
early detection and prompt 
treatment of illness have 
demonstrated that mass media 
and other communication 
strategies can be effective in 
reducing the risk of serious 
illness. 

Communicating effectively 
about health is a difficult task. 
Health information is often 
connplex and technical. In 
addition, the information may be 
inconclusive, controversial, 
contradictory and subject to 
change as new research 
findings are released. Many 
diseases such as cancer are 
fear-arousing; individual 
responses may be emotional. 
New health information may 
conflict with long-held personal 
belled. As a result, the potential 
exists for misdirecting or 
alienating the public. 



Careful planning and de- 
velopment of health communi- 
cation programs are important to 
avoid these undesirable effects, 
and to assure that communica- 
tion activities have the greatest 
potential for success. 

For more than a decade the 
Office of Cancer Communica- 
tions (OCC), National Cancer 
Institute, has been developing 
communication messages and 
programs for health profession- 
als, patients, and the public. We 
have learned a great deal about 
how to develop effective com- 
munications. Over the years we 
have received many requests for 
assistance in planning programs 
—this guide is largely directed 
to the questions we have been 
asked. 

The purpose of this manual is 
to learn from and share our 
experiences, and those of others 
—those who are faced with 
planning health communication 
programs. This book discusses 
some key principles relative to 
specific steps in program devel- 
opment, and includes examples 
of their usa Sources of addition- 
al information on each subject 
are included at the end of the 
chapters. A glossary, a bibli- 
ography, and other resources 
can be found in the appendices. 



This guide expands upon and 
replaces "Pretesting in Health 
Communicetions:' Information 
about pretesting is incorporated 
into this book; in addition, we 
hope that this new text will help 
explain the factors beyond ma- 
terials pretesting that contribute 
to a successful health communi- 
cation program. 

To request additional copies 
of the guide please contact: 

Rose Mary Romano 
Office of Cancer 

Communications 
National Cancer Institute; 
Building 31, Room 4B43 
Bethesda, MD 20892 
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This book is designed to 
help professionals in 
health and health-related 
agencies communicate with the 
public. It offers guidance for 
planning a health communica- 
tion program based on social 
marketing and other principles 
and the experiences of National 
Cancer Institute staff and other 
practitioners. It Is designed to 
help yoLi identify what to do and 
why — and where to go for 
assistance. 



An overview of each step in 
program development is in- 
cluded in the section entitled 
"The Health Communications 
Process" (pages 5-6). This 
section has been designed so 
that you can duplicate and share 
it, use it to substantiate your 
program plans, or to increase 
the awareness of your own staff. 

Each chapter subheading lists 
what is covered within — to help 
you select those sections that 
relate to a specific task you face. 
Checklists and planning ques- 
tions are included to help you 
shape your tasks, and the 
selected readings at the end of 
each chapter direct you to more 
complete infornnation about 
specific subjects. The Appen- 
dices include a glossary and 
other sources of planning and 
health-related information. 



Sample forms, questionnaires, 
and a theater test methodology 
are also included to help you in 
developing your own pretests. 

We request suggestions you 
may have for future editions of 
this guide; a comment form is 
on the last page of this guide. 
We welcome your comments. 
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In 1980 the 
Department of Health 
and Human Services 
published the 
Objectives for the 
Nation: 7990. These 
health objectives, 
compiled by 
groups of experts addressing 15 
priority areas, projected an 
improvement in our national health 
status that could be reached by 1990 if 
current knowledge were to be applied. 
Better use of existing health 
knowledge requires communication 
among health care and social service 
professionals, related organizations, 
government agencies, the private 
sector, and individual citizens. In fact, 
communication between leaders in 
health, education, industry labor, 
community organizations and others 
was cited as essential in reaching the 
1990 objectives. Health 
communication programs can be 
designed to inform, influence and 
motivate institutional or public 
audiences. 
Communication can: 

o increase awareness of a hea'th 
issue, problem or solution 

□ affect attitudes to create support for 
individual or collective action 

□ demonstrate or illustrat ^ skills 

□ increase demand for health services 

□ remind about or reinforce 
knowledge, attitudes or behavior. 



Health communication programs 
cannot: 

□ compensate for a lack of health care 
services 

□ produce behavior change without 
supportive program components 

□ be equally effective in addressing all 
issues or relaying all messages. 

Therefore, communication should be 
included as one component in pro- 
grams designed to address a health 
problem. 




The National Cancer Institute has 
incorporated aspects of various com- 
munication models, theories and 
practicey into its planning process. 
Each discipline offers a different 
perspective on the consumer and the 
steps leading to behavior change. F=br 
example, social marketing practice 
considers the perceptions and per- 
ceived needs of the target audiences 
as an essential element of planning. 
Health education models involve an 
exploration of the components of 
behavioral intention that will influence 
an individuars willingness to act. Mass 
communication theories help explain 
factors that influence message trans- 
mission between the source and the 
target audience and the expected 
effects. Summarized below are some 
of the theories, models and practices 
that have been applied in developing 
this guide. 

Social M«1(«tin9 

Social marketing models, first articu- 
lated by Philip Kotler and based on 
commercial marketing practices, show 
that the consumer (target audience) 



should be the central focus for 
planning and conducting a program. 
The program's components focus on 
the: 

□ pr/ce— what the consumer must 
give up in order to receive the 
program's benefits. These ••costs" 
may be intangible (eg., changes in 
beliefs or habits) or tangible (eg., 
money, time or travel) 

□ procfucf— what the program is trying 
to change within the target audience 

n promotion— ^o^ the exchange is 
communicated (eg., appeals used) 

□ p/ace— what channels the program 
uses to reach the target audience 
(e.g.. mass media, community, 
interpersonal) 

The formulation of price, product. 
pronfX)tion and place evolves from re- 
search with the consumer to deter- 
mine what benefits and ••costs'* they 
would consider acceptable, and how 
they might be reached. Lessons 
learned from social marketing stress 
the importance of understanding the 
target a dience and designing strate- 
gies based on their wants and needs 
rather than what good health practice 
directs that they ••should" do. 

Behavioral IfitMitioitf 

Studies of behavioral intentions sug- 
gest that the likelihood of the target 
audience adopting a desired behavior 
can be predicted by assessing (and 
suDsequently trying to change or 
influence) their attitudes toward and 
perceptions of benefits of the behavior, 
along with how they think that their 
peers will view their behavior. Re- 
search by Fishbein and Ajzen support 
the idea that an individuars. and 
society's (perceived) attitudes are an 
important predecessor to action. 
Therefore, an important step toward 
influencing behavior is a preliminary 
assessment of target audience atti- 
tudes, and subsequent tracking to 
identify any attitudinal changes. 
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Communications for Ponuation 

William McGuire has described the 
steps an individual must be persuaded 
to pass through in order to assimilate a 
desired behavior. These steps are: 

□ exposure to the message 

□ attention to the message 

□ interest in or personal relevance of 
the message 

□ understanding of the message 

□ personalizing the behavior to fit 
one's life 

□ accepting the change 

□ remembering the message and 
continuing to agree with it 

□ being able to think of it 

□ making decisions based on bringing 
the message to mind 

□ behaving as decided 

□ receiving (positive) reinforcement for 
behavior 

□ accepting the behavior into one's life 

To communicate the message suc- 
cessfully, five communication compo- 
nents all must work: 

□ the credibility of the message 
source 

□ the message design 

□ the delivery channel 

□ the target audience 

□ the targeted behavior 

Attention to McGuire's considera- 
tions helps assure that the communi- 
cation program plan addresses all 
fdotors that determine whether a 
message is received and absorbed, 
and that the program is staged over 
time to address audience needs as 
they differ over time while progressing 
toward behavior change. 



DHfuoion of Innovationt 

The health policy makers call It **tech- 
nology transfer;'* Everett Rogers 
describes the process whereby new 
products or ideas are introduced or 
"diffused" to an audience. Whether 
the message is accepted (or the 
behavior adopted) depends upon 
whether the audience: 

□ perceives it as beneficial 

□ sees it as in accordance with their 
needs and values 

□ finds it easy or difficult to under- 
stand or adopt 

□ can try the behavior 

□ feels that the results of the trial or 
acceptance are viewed positively by 
their peers. 

Rogers suggests that the mass 
media are a quick and effective route 
for introducing new information or 
trying to influence attitudes, especially 
in the early stages of reaching audi- 
ences predisposed toward accepting 
new ideas. However, at the point of 
trial, or **adoption.'* interpersonal 
channels are more influential. This 
means that a communications strategy 
might consist of using the mass media 
to introduce the message, provide 
knowledge, influence attitudes, and 
reinforce behavior, and using commu- 
nity or interpersonal intervention to 
teach and encourage the adoption of 
the behavior. This is especially 
important for groups known not to 
adopt new behaviors quickly 

PRECEDE Modol 

Lawrence Green developed the 
PRECEDE model, an approach to 
planning that examines the factors 
which contribute to behavior change 
These include: 

□ predisposing factors— the indi- 
vidual's knowledge, attitudes, 
behavior beliefs, and values prior to 
intervention that affect their willing- 
ness to change 



□ enabling factors — the structure of 
the environment or community and 
an individual's situation that facilitate 
or present obstacles to change 

□ reinforcing factors— the positive or 
negative effects of adopting the 
behavior (including social support) 
that influence continuing the 
behavior. 

These factors require that the 
individual be considered in the context 
of their community and social struc* 
tures. and not in isolation, when 
planning communication or health 
education strategies. 

Understanding these concepts 
related to health communication 
planning can help assure a successful 
program. A more thorough review may 
be valuable for planning a particular 
program. Sources of more information 
include: 

r ^hbein. Martin and Ajzen. I . Belief, 
Attitude, Intention and Behavior, Reading. 
MA: Addison-Wesley. 1975. 

Green, Lawrence W.. Kreuter. Marshall W. 
Deeds. Sigrid. G.. and Patrtdge. Kay B.. 
Health Education Planning, A Diagnostic 
Approach, Palo Alio. Mayfield Publishing 
Company, 1980. 

Kotler. Philip and Andreasen. Alan R,. 
Strategic Marketing for Nonprofit 
Organizations, 3rd Edition. Englewood 
Cliffs. NJ: Prentice-Ha'il. 1987. 

Kotler Philip. Ferrell. OC. and Lamb. 
Charles. Strategic Marketing tor Nonprofit 
Organizations: Cases and Readings, 
Englewood Cliffs. NJ: Prentice-Hall. 1987 

McGuire. William. "Theoretical 
Foundations of Campaigns." in Rice. R.E . 
and Paisley W.J. (Eds.). Public 
Communication Campaigns, Beverly Hills 
Sage Publications. 1981. pp. 41-70. 

McOuail. Denis and WendahL Sven. 
Communications Models for the Study of 
Mass Communications. New York: 
Longman. Inc . 1981. 

Rogers. Everett. M.. Diffusion of 
Innovations, New York: Free Press. 1983 

Additional sources are included in 
Appendix F (Bibliography) 
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While it is true that there are spe- 
ciaiists who use sophisticated tech- 
niques to evaluate progmms, it is also 
true that evaluation is a natural 
process. We all take actions that are 
assessed by consciously or uncon- 
sciously reviewing the available facts, 
considering them in the light of the 
original intent, and drawing a 
conclusion. 

For example, you might find that the 
news media rarely report your news as 
you think they should. A close look at 
the situatkxi— the content of your 
news releases, how and when they are 
released, and reactions of reporters 
receiving them— might identify and 
lead to solving the problem. The 
purpose of any evaluation is to learn 
from actions so that improvements 
can be made. 
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Everyone reaches conclusions 
about the relative success or failure of 
programs and activities. Formal eval- 
uation helps assure that those con- 
clustons are based on objective data. 



Formal evaluation takes the natural 
process and awakes it a conscious, 
orderly effort, using objective tech- 
nk]ues for gathering and analyzing 
data and reaching conclusions in order 
to: 

□ improve current and future efforts 

□ ceitify the degree of change that 
has occurred 

□ identify programs, or elements ot 
programs, that are not working. 

Evaluatkxi is one of nr^ny tools 
available to help communicatton 
program planners and other decision- 
makers do their jobs well. However, it 
is important to recognize that there are 
many forms and types of evaluation, 
from the very informal and simple to 
the very fbmial and complex. 

Evaluatton is not a task that is 
tacked onto the end of a program. 
Assessment and careful planning are 
interdependent, integral functkKis of 
program developnnent and implenr>en- 
tation. Just as each step of a program 
contributes to its effect, each step can 
be subjected to evaluation. Even 
before program development begins, 
evaluative discipline demands that the 
desired program outcome be de- 
scribed as specifically as possible 
Once set, these goals and objectives 
direct how each aspect of the program 
will be d&/e\ooed. In this guk^ evalua- 
tion strategies are incorporated into 
each stage of program devebpment. 
In addition, a more conriplete discus- 
sion of program evaluation is included 
in Stage 5: Assessing Effecti\/eness. 
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Some exanrvles cited in this workbook 
are large-ocale natkKiwkle programs; 
others were developed for a state or 
community. Most of the planning steps 
and consWeratkxis apply regardless of 
geographic span. Similarly, soa\e 
health problenr>s are nwe complex 
and controversial than others. They 
may i^rvolve a select or nrwiny diverse 
taiyet audiences; simple infbnrnatkxi or 
complex behavtor change; one or 
many channels; generous budgets 
over nrwiny years, or almost no budget. 

This guide describes what we have 
found to be a practical scheme for 
planning and implementing health 
communication programs. However, 
you nrwiy encounter a situatkxi that 
does not permit or require each step 
outlined here We hope you will 
consider each issue and step included 
and make a conscious decision 
regarding whether it applies to your 
situatton. Clearly, there are no hard- 
and-fast rules; consider the following 
chapters as they are Intended— as 
suggested guidelines. 
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A fundamental premise of this 
workbook is that to be viable, 
health communicatkxi programs 
must be based on an understanding of 
the needs and perceptk)ns of their 
target audtences. The diagram belovv' 
illustrates an approach to h.ealth 
communicatkxi incorporating assess- 
menis of target audience needs and 
perr;eptk>ns at critical points in 
program devetopment and implemen* 
tatkx). The six stages constitute a 
circular process, in whteh the last 
stage feeds back to the first in a 
continuous process of planning and 
inriproverDent. 

Each of the six stages is sum* 
marized in this chapter to provide you 
with an overview c* this approach, 
then in nxxe detail in the following 
chapters. The steps outlined befow 
constitute an kieal process, one that 
may require more time and nx)ney 
than many agencies can afford. All of 
the steps may not be feasible, or in 
some cases even essential. However, 
carefully folfowing the steps in each 
stage of the process can make the 



next program phate vnore productive 
In general, however, you must apply 
your professkxial judgment to deckle 
which steps ^e appropriate for your 
particular program. 




The planning stage of a program 
provkles the foundatk)n for the entire 
health communicatton process, faulty 
deciston making at this point can lead 
to the development of a program that 
is "off the mark." Careful assessment 
of a problem in the beginning can 
reduce the need for costly midcourse 
corrections. 

KtylssuM 

■ What is already known about the 
health problem? (Analyze existing 
data.) 



Figure 1 

SIMM in HmWi Commun lM tkm 




■ Stages when pretesting is used. 



■ What new kinds of infonmatfon will 
be needed before planning the 
prograrrt? (Generate new data if 
needed.) 

■ Who is the target audience? What is 
known about them? 

■ Overall, what change is planned to 
soive or lessen the problem? (goals) 

■ What measurable objectives can be 
established to define success? 

■ How can progress be measured? 
(Ran evaluation strategies.) 

■ What should the target audience be 
told? (Draft communication 
strategies.) 




The decisions you make in stage 1 will 
guide you in selecting the appropriate 
communication channeKs) and pro- 
ducing effective materials. Without 
clear objectives and a knowledge of 
your targpt audience, you risk pro- 
ducing 'naterials which are inappro- 
priate for the target audience or the 
issue being addressed. 

KeyltsuM 

■ Are there any existing materials 
which could be adapted for the 
progranrt? 

■ Which channels are most appro- 
priate for reaching the target 
audience? (eg., worksite, mass 
media, face-to-face) 

■ What materials formats will best suit 
the channels and the messages? 
(e.g., booklets, videotapes, curricula) 



In stages 1 and 2 most program 
planning is completed; this planning 
provides the basis fnr developing 
messages and materials. Often several 
different concepts will be developed 
and tested with target audiences. 
Feedback from the intended audience 
is critical in stage 3. 

■ What are the different ways that the 
niessage can be presented? 

■ How does the target audience react 
to the nriessage concepl(s)? 

■ Does the audience: 

■ Understand the messav,^? 

■ Recall it? 

■ Accept its importance? 

■ Agree with the value of the 

solution? 

a How does the audience respond to 
the message format? 

■ Based on responses from the target 
audience, do changes need to be 
made in ttie message or its format? 

■ How could the message be pro- 
moted, the materials distributed, and 
progress tracked? 



The fully devetoped program is 
introduced to the target audience; 
promotkKi and distributkni begin 
through all c^annels, Program com- 
ponents are oerkxlically reviewed and 
revised if necessary. Audience 
exposure and reaction are tracked to 
permit alterations if needed. 

■ Is the message making it through 
the Intended channels of communi- 
catton? 

« Is the target audience paying atten- 
tion, and reacting? 

■ Do any existing channels need to be 
replaced, or new channels added? 

■ Which aspects of the program are 
having the strongest effect? 

■ Do changes need to be made to 
improve program effect? 



At each stage useful infemnatkxi was 
gathered about the auo'^ence; the mes- 
sage, the channels of communication 
and the program's intended effect. All 
of this infonT)atk}n helps prepare for a 
new cycle of program de^ekyprrtevX. 
The more information that can be 
reviewed at the end of the first 
program phas^ ♦he rr^xe likely it is 
that these questtons can be answered: 

■ Why dkJ the program work, or not 
work? 

■ Are there program changes or inrv 
provements that should be made to 
increase the likelihoou of stiocess or 
to address change In the audience, 
or problem or other situations? 

■ Are there lessons learned that could 
help make future programs more 
successful? 



The program should be assessed by 
analyzing the results of measurements 
planned in stage 1 and used through- 
out the program's lifespan. 

ItoyltsuM 

■ Were the program objectives met? 

■ Were the changes which took place 
the result of the program, other 
factors or a combination of both? 

■ How well was each stage of pro- 
gram planning, implementation and 
assessment handled? 
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Planning and Strategy Selection 




The first purpose of planning is to 
detenmine whether the problem 
can be addressed through conn- 
munication. If it can be» the planning 
stage provides the foundation for the 
entire health communication process. 
Flawed decision making at this point 
can lead to the development of a 
comnnunication program that is "off 
the mark!* Careful assessment of a 
problem in the beginning can reduce 
the need for costly midcourse 
corrections. 

WiMra Do You Start? 

When faced with developing a new 
comnnunication program, it may seem 
as though you must do everything at 
onca To organize your thoughts about 
planning, ask yourself questions like 
these: 

B What is the health problem to be 
addressed? 

□ Who is affected by it, and how? 

Q Are they aware that the problem 
could affect them? 

0 Who is interested in the problem? 

0 What activities have been 
addressing the problem? 

0 Are the media or other organizations 
doing anything? 

B What can we say or do to help 
anneliorate the problem? 

S3 To whom? 

E3 What do we want to accomplish? 

B What resources are available*^ 

This chapter is designed to help you 
find the answers to these questions 
and design a program plan. 

Revivw Availeble Data 

(Secondary Research) 

The more you understand about an 
issue or health problem the better you 
can target corrective measures. Check 
all potential sources of information in 
your agency; identify gaps in what you 
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have, and seek outside sources of in- 
formation. The types of information you 
should (ideally) have to plan your 
communications program include: 

0 a description of the problem 

□ incidence 

□ effects on individual and 
community 

o causes arid preventive measures 

□ solutions, treatments or remedies 

B a description of who is affected 
(potential target audiences) 

□ age, sex. ethnicity, places of work 
and residence 

□ causative/preventive behaviors 

□ related knowledge, attitudes and 
behaviors 

□ patterns of use of health-related 
services 

□ media preferences and habits 

□ information sources that are 
considered credible by potential 
target audiences 

The purpose of this data collecUon 
is to describe the health problem or 
issue, who is affected, and what they 
know, believe, and do. This investiga- 
tion should also help identify related 
activities. 

Sources of information will, of 
course, vary by issue and by wheth'^r 
your program will relate to a specific 
community or broader geographic 
area. Examples of data sources 
include: 

@ library searches 

@ sources of health statistics (a local 
hospital, a state health department, 
the National Center for Health 
Statistics) 



0 government agencies, universities, 
voluntary and health professional 
organizations 

a clearinghouses 

□ advertising agencies, newspapers, 
radio and television stations (for 
media use data, buying and con- 
sumption patterns) 

C3 community service agencies (for 
related service use data) 

□ corporations (e.g.. General Mills), 
trade associations and foundations 

^ polling companies (for audience 
knowledge and attitudes) 

0 depositories of polling information 
(e.g.. the Roper Center, University of 
Connecticut) 

0 local Chambers Oi Commerce 

Both published and unpublished 
reports may be available from these 
sources. 

A number of Federal health infor- 
mation clearinghouses provide infor- 
mation, products, materials and 
sources of further assistance for 
specific health subjects. A helpful first 
step in planning may be to contact the 
appropriate clearinghouse and your 
health department to obtain informa- 
tion on the health issue A clearing- 
house referral source and other 
potential sources of information are 
included in appendix A. 

Identify Existing Activities 
and Gaps 

If another organization ii? already ad- 
dressing the problem, you may want to 
contact them to discuss* 

D what they have learned 

■ what information or advice they may 
have to help you plan 

B what else is needed (what gaps 
exist) 

B opportunities for cooperative 
ventures. 
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Write Qoalt ami ObJecUvts 

The gcals and objectives establish the 
shape of the program — what it is de- 
signed to accomplish. The program 
goal or goals describe the overall 
change (e.g., a specific improvement in 
one aspect of the health of a certain 
population). Reaching the goal may 
encompass service delivery, financial 
and societal support and other 
situational changes tha^ can only be 
partially addressed by communication 
strategies. The Office of Disease 
Prevention and Health Promotion's 
Promoting Health/Pre\fvnting Disease 
(the 1990 Health Objectives for the 
Nation) can help you set goals for 
some health issues. 

Objectives describe the inter- 
mediate steps that must be reached to 
accomplish the broader goals; th^ 
describe the desired outcome, but not 
the steps involved in obtaining it (you'll 
design strategies for getting there 
later). Some of these objectives may 
be beyond the scope of a communica- 
tion program; this guide only ad- 
dresses translating the communication 
objectives into action. 

Objectives are written to articulate 
what your program is intended to do. 
Therefore, communication objectives 
should be: 

B specific 

B attainable 

■ prioritized to direct the allocation of 
resources 

Q measurable to assess progress 
towards the goal 

a time specific 

These objectives are the foundation 
for program development and evalua- 
tion. If they are not clear and "action- 
able;* your program may be unfocused 
and ineffective. Once written, goals 
and objectives serve as a kind of 
program "contract," or agreement 
regarding program purpose. 



For example: 



7b Increase by 1Q percent the 
numt)efMpedple\y^ 
hlgh^ blotXiik^ 1991) 

To (tet^aridr^ttopeet^ 
IrnMiuals with elevated 
_bb(xi pressure. 



For this objective, a communications 
activity might include a campaign to 
tell middle-aged blanks (or another 
designated high risk group) where to 
go for screening, and motivate them to 
go. If screening services are not 
conveniently available, a critical first 
step would be to identify whether and 
where screening, referral and patient 
followup services could be made 
available. 
Another example: 



QMb 

To reduce ty 5 percent the 
nurhber of parents of young 
children who smoke ty 1991. 

To Increase by x percent the 
number of pediatricians who 

counsel their patients* parents 
about how to quit smoking 

by 1991. 



For this objective, it would be 
necessary to measure how many 
pediatricians currently counsel these 
parents, and establish a numerical 
objective based on your resources and 
realistic expectations for change 
Strategies might include providing 
pediatricians with the latest infor- 
mation about the effects of parental 
smoking on their children, designing 
counseling guidelines for pediatricians, 
and encouraging parents to ask their 
children's pediatrician for help in 
quitting. 



Goals and objectives should be 
realistic >bu don't want your program 
to be considered a "failure" because 
you set unrealistic expectations. For 
example, it is generally impossible to 
achieve a goal of 100 percent. An 
epidemiologist or statistician may help 
you determine recent rates of change 
related to your issue so that you have 
some guidance for determining what 
additional changes could be affected. 

Gather New Data 

(Primary Research) 

You may find that there is not enough 
known about the health problem, its 
resolution, or those who are affected 
to develop a communication strategy 
If the health problem cannot be de- 
fined, or there is no way to ameliorate 
it, you may decide wisely that a com- 
munication program is an inappro- 
priate response until more information 
or appropriate actions become 
available. 

If it appears that there is no action 
an individual can take because the 
issue is beyond individual control (e.g.. 
the regulation of the use of toxic 
substances), not subject to collective 
action, services are not available (eg., 
for a high risk, low income population), 
or there is no treatment, you must 
decide whether a communication pro- 
gram should be developed. In some 
cases, you may need to make sure 
that other program elements are in 
place prior to developing a commu- 
nication program. (However, in this 
case, you could decide to develop 
communication strategies directed to 
policymakers to seek their help in 
defining and addressing the problem.) 

More often, you can define the 
problem and who is affected, but 
information about that population may 
be unavailable or outdated. At this 
point, you may decide to gather new 
data before planning a communication 
program. This investigation can involve 
a probability sample of respondents 
with results which can be projected to 
the whole target audience— as well as 
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its subparts— (quanf/faf/ve research), 
or fewer representatives typical of the 
group to be reached but with a less 
stringently designed sample (quanta- 
f/ve research). Or it may fall some- 
where in between— using a combina- 
tion of research methods (eg., focus 
groups along with a small-scale tele- 
phone survey). Agencies frequently 
rely on qualitative research because it 
is faster and less expensive 

Quantitative research such as 
surveys with large, statistically 
representative samples usually use 
structured questions administered 
through: 

H personal interviews (at home or 
intercepted at another location) 

a telephone 

@ mail 

Quantitative research may also take 
other forms such as collecting vital 
statistics data. This process can 
estimate what percentage of a 
population is aware of the issue, or 
thinks or behaves in a certain way. 
These data provide: 

@ a valuable baseline for tracking 
changes as a result of your 
communication intervention 

B information for setting priorities (eg., 
according to which groups of people 
seem to lack awareness, or are 
most likely to behave in an unhealthy 
manner) 

0 information about segments within a 
target population who are more in 
need of attention. 

Usually, quantitative data will not 
reveal why individuals think or act as 
they do; also, because information is 
available only in response to specific 
questions asked, unexpected factors 
or influences may not be identified. 

Qualitative investigation can provide 
the exploration into the "why." 
Examples of qualitative studies 
Include: 

■ focused group discussions ("focus 
groups") 



B a small number of "open-ended" or 
in-depth interviews. 

These and other methods are 
discussed in Stage 3. These methods 
can provide: 

0 information about target audience 
perceptions 

B identification by the target audience 
of issues they perceive as related 
and important 

B in-depth discussion of what the 
particular health issue means to the 
target audience. 

Although qualitative research can 
help identify and explore issues, 
because of the small numbers of 
respondents and the lack of stringent 
sampling, the findings cannot be 
projected to a population as a whole. 

DttermiiM Your Target 
Audiences 

specifically describing the audience 
(or audiences) for your program— who 
you want to reach and influence with 
your messages — will help you develop 
relevant messages and materials and 
identify the channels most likely to 
reach them. Few messages are appro- 



priate for everyone included in the 
"general public," given the diverse 
interests, needs, concerns and 
priorities among different segments of 
the public. Trying to reach everyone 
with one message or strategy may 
dilute your message so that it appeals 
to few rather than many people 

Describing Your TirBtl AinHmc* 

Try to think about all of the physical, 
demographic (and, perhaps, psycho- 
graphic) characteristics of the people 
you are trying to reach to help divide 
"the public" into more manageable 
groups or target audiences. 

These characteristics include: 
a physical— sex, age, type and degree 
of exposure to health risks, medical 
condition, disorders and illnesses, 
health history of family 
a tieftaWora/— media exposure, 
membership in organizations, health- 
related activities or actions and 
other lifestyle characteristics 

a demograpWc— occupation, income, 
educational attainment, family 
situation, places of residence and 
work, cultural characteristics 
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^ psychographic — attitudes, opinions, 
beliefs, values, self-appraisal and 
other personality traits 

The more complete a "profile" or 
description you can develop of your 
audience, the better you will be 
prepared to develop a program suited 
to them. 

"StgnMfitiiig" Tirgitt AudlMCtt 

Once you describe what you know 
about who you want to reach, you 
should be able to "segment" or 
separate your target audience from the 
rest of the general population. You may 
find that you want to reach several 
distinct population groups. 




Primary target audiences are those 
you want to affect in some way; you 
may have several primary target 
audiences. If so, you should set 
priorities among them to help order 
your planning and allocate your 
resources. Secondary target 
audiences are those with influence on 
the primary audience or those who 
must do something in order to help 
cause the change in the primary target 
audience 

The process of identifying and 
defining audiences should lead to 
setting audience priorities, that is: 

0 deciding which one (or several) 
audience segments is most 
important 



Q deciding which audiences are 
important but less critical because 
of their health risk/health status, 
influence or link to the primary target 
audience(s) or limited program 
resources 

B deciding who will not be a target 
audience for the progrann. This deci- 
sion provides valuable program 
direction for decisions regarding 
message development and dissemi- 
nation, helping assure that all pro- 
gram resources are spent 
productively 

Establish Audisnce Ihicking 
System 

If you are designing a major communi- 
cation program that will stretch over a 
long period of time, building audience 
tracking into your program plans will 
assure that you can: 

B find out what your target audience 
knows, thinks and does before you 
begin activities (baseline data) 

s periodically survey to assess pro- 
gress and the need for modification 
or new activities 

B identify the change in status among 
the target audience when your pro- 
gram is completed. 

All too frequently audience surveys 
only are undertaken during and after 
the program, or are inappropriately 



timed to occur too far after the pro- 
gram completion, or are sporadic, or 
incompatible and results cannot be 
compared. To avoid these problems, 
consider whether audience tracking is 
appropriate for your program at this 
early planning stage. 

Assess Resources 

It is important that you assess your 
resources to determine what and how 
much you realistically will be able to 
accomplish. Setting realistic expecta- 
tions can help you avoid the frustration 
of not accomplishing as much as 
anticipated. To set realistic objectives, 
think about these questions: 

1. What are the greatest areas of 
need? 

2. Which activities will contribute the 
most to answering these needs? 

3. What resources are available? 
Include: 

□ staff and other "people" re- 
sources—such as committee 
members, associates from other 
programs, and volunteers 

□ budget— funds and "in kind" 
resources such as computer 
time, mailing costs, printing 
services available from another 
source, educational materials 
free or at cost 
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0 information— about the issue, the 
target audience, the connnnunity 
and media structures, or about 
available educational materials 

0 time — that is, how many weeks, 
months or years are available to 
complete the program. 

^ What community activities, organi- 
zations or other contributing factors 
<ist? 

. v\/hat barriers (such as approval 
obstacles, absence of funding, 
hard-to-reach target audience) are 
there? 

6. Which activities would best utilize 
the resources you have identified 
and best tit within the identified 
constraints? 

Those activities identified in question 6 
should become your priorities. 

Coping wKh Limited RMourMt 

"Resources" includes a lot more than 
funding, as demonstrated in question 3 
above. Sometimes you may feel so 
constrained by a lack of funds that 
developing a program appears impos- 
sible. An honest assessment may lead 
to the conclusion that a productive 
program is not possible; on the other 
hand, sufficient intangible resources 
(that is, other than "hard funding") 
may be available to proceed. 

Remember: adequate funding alone 
won't guarantee program success. In 
addition to careful program devel- 
opment, you'll need the cooperation 
and help of your own associates and, 
perhaps, other organizations. That's 
why questions 5 and 6 above are 
important to consider as you decide 
whether and how to develop a new 
health communication program. 

Draft Communication 
Stratagios 

Now that you have defined what needs 
to be done (goals and objectives) with 
whom (target audiences), it is- time to 
design communication strategies most 
likely to get you there. The strategy 
statement begins with: 
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■ the program objectives 

■ the primary and secondary 
audiences, 

and adds: 

■ the target information to be 
communicated 

■ benefit, as perceived by audience. 

The strategy statement provides all 
program staff — including writers and 
creative staff— with the same direction 
for developing all messages and 
materials. It also may contain the 
tactics that will be used to reach target 
audiences with the appropriate 
messages. 

The audience benefit— what they 
will gain that they perceive as im- 
portant or valuable— may be different 
from your percepiions of what the 
benefit is (eg., ir ^proved health) — and 
may be different )or each audience 
identified. Stage 3 discusses focus 
groups and other means to help 
identify what the target audience 
perceives as important. 

The strategy statement also should 
describe w/7y the audience would want 
the benefit, to help direct creative 
develoDment. 




Developing the strategy statement 
provides a good test of whether you 
have enough information to begin 
developing messages. >bu may be 
tempted to skip this step, but the 
strategy statement forms a foundation 
and the boundaries for all creative 
development. Seeking agency (and 
perhaps, community) approval of the 
strategies at this early stage can make 
them feel informed and ease the 
approvals and cooperation you may 
need later. 

Once you have decided on the com- 
munication strategies, all program 
elements should be compatible v\^th 
these strategies— that means every 
program task should contribute to the 
established objectives, and be tar- 
geted to the identified audiences; all 
messages and materials should 
incorporate the benefits and other 
information in the strategy statement. 




As you develop the program and 
learn more about the audience and 
their perceptions, you may need to 
alter or refine the strategy statement. 
However, it only should be changed to 
reflect imoroved information that will 
strengthen your capability to reach the 
program's goals. It should not be 
altered to accommodate a great idea 
that is "off strategy." 
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Qoal 

Planning Tasks Person Responsible Due Date Resources Required 



Implementing Tasks Person Responsible Due Date Resources Required 



Evaluatk>n Tasks Person Responsible Due Date Resources Required 



Write Program Plan and 
Timetable 

All of the elements of your planning 
should be recorded in a program 
plan— your first program product. The 
program plan is your "blueprint.'* 11 
should be used to: 

■ design all program tasks 

■ explain your plans within your 
agency and with ethers 

■ provide a record of where you 
began. 

A sample outline for a program plan is 
included on page 15. 

You will note t!iat the program plan 
outline includes your evaluation plans. 
Because evaluation occurs at many 
points— before, during and after the 
program— descriptions of evaluative 
strategies are dispersed throughout 
this workbook, within the stages where 
the evaluation would occur. Formative 
evaluation is discussed in stages 1 
and 3, process evaluation in stage 4 
and summative evaluation in stage 5. 
A summary of the various kinds of 
evaluation can be found on page 64 
(stage 5). 

Program Timttabto 

A final planning step is producing a 
time schedule for program devel- 
opment and implementation. The 
schedule should include every task 
you can think of from the time you 
write the plan until the time you intend 
to complete the program. The more 
tasks you build into the timetable now. 
the more likely you will remember to 
assign the work, and keep on 
schedule If you forget important 
intermediate steps, your costs and 
timing might change. The timetable 
could look li'<ethis: 



0blectl¥9 2: etc. 

Of course, nothing ever goes quite 
according to plans! The timetable 
should be considered a flexible 
management tool. MjU may want to 
update it regularly (e.g., once a month) 
so that it can function dually to 
manage and track progress. 
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DMorlpUon of NmcI: (why the prograrr i$ being 
developed) 



OliMM' I totouro w Itoqiiiradi (e^., staff, art shop or 
computer tlrhe),^ 



PrinuNif Jmrg0l AMcMno#t« (in priority order) 
(Include age, gender, ethnic group and other 



pertinent characteristics). 



I (scheduling conflicts, 
clearances, policies and approvals that you and other 

staff must address) 



K«y t t r<t e gl #t> (list for each target audience) 



t eoo n Jary 1hrg«t AuAmomi (in priority order) 



M tth o ds off IvahMrtlons (include fbmiattve, 
process and sunrmttve (outcome) evaluation strategies) 
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1. State program goat 



a) What are the modifiable 
risk factors? 

b) What risk fectors/issues are 
the fiilghest priority for 
program development? 



Agency misston statenrient 
Role of communicatfons 



Literature searches 
Epidemiologic studies 
Consultation with experts 
Agency policies 



Prevention: 

To reduce the incidence of cancer 
through informatten and education 
programs that promote publto 
adoption of risk-reducing behavfors. 

SnK>king 
Diet 



S. a) What are the behaviors 

associated with the identified 
risk factors/issues? 
b) Whichbehavtors are high 
priority for intervention? 

4. What audiences should be 
targeted? 



Literature searches 
Epkiemidogic studies 
Consultation with experts 
Agency policies 

Review of mort)idity/noortality 
statistics* 

Review of knowledge/attitudes/ 
behavfor (KAB) sun/eys 

Agency policies and politics 

Review of other Federal programs 

Consultation with experts (advisory 
groups, wori(ing groups) 

Secondary data tracking 



Prevention: 

SnrK)king; fat and fiber intake 



adults 

feimily caretakers 
low-SES blacks 
Hispanics 



5. What are the key factors 
influencing the behaviors in the 
target audience? (KAB, ^ills, 
availability of resources, 
professional attitudes) 



e. Develop and state program 
objectives. 



Literature reviews 

Consultatton with experts 

Focus groups (audience members, 
professionals) 

Psychographic surveys 

NCI surveys 

Secondary data tracking 

Literature reviews 

Consultation with experts 

Focus groups (audience members, 
professionals) 

Psychographic surveys 

NCI surveys 

Secondary data tracking 



Cancer Prevention in iow-SES blacks 
—fear and avoklance of cancer 
(attitude) 

^medical care is not effective (belieO 
—smoking cessation, food selection 
skills 



Smoking 

—increase the number of health 
professionals who counsel their 
patients to quit smoking 

—increase the number of national/ 
regfonal organizations that sponsor 
tobacco education programs 



7. VVtwt art the bast Channels to 
uselo reach the taroet « 
autfence? (mediat VMxk^ 
national and community 
voluntary organizations, 
churches, the medical sector) 



a. a) What is the t)e8t program/ 
products to use in the selec- 
ted channel with the target 
audience/? (Define messages, 
and strategy; develop prod- 
ucts to support them.) 

b) Are the selected products 
clear and effective to the 
intermediary? 1b the target 
audience? 

c) Are program/lproducts likely 
to produce the desired 
outcome with the inter- 
mecfery? %rget audience? 

e. How will the program/product be 
distributed/promoted? 



10. Develop an action plan with 
staff and resource allocations, 
timetables, expected outcomes. 



11. a) is the product reaching the 
intermediary? 

b) is the intermediary using the 
product with the target 
audience? 

c) What percentage of the 
target audience Is being 
reached? 

d) Is the product producing the 
desired outcome with th^ tar- 
get audience (e^., calls, at- 
tendance at presentations)? 

It. Assess results of tracidng and 
evaluations. What changes are 
indicated? 



Analysis of the reach and influence of 
the channel on the target audience 
through:; 

—literature reviews 
—review of other programs 

targeting the audience 
—focus groups 
—consultation with experts 
Assessment of resources/role (staff, 
budget) 

Agency policies and politics 

a) Resources 
Past successes 
Concept testing 
Focus groups 
Consultation with experts 

b) Pretests 
Readability tests 



c) Review of evaluations of similar 
programs/products 
Pilot tests 



Analysis of various interiDediary 
networi($ 

Review of past NCI programs 
Review of other comparable 
programs 



Blacks 

—black newspapers and nmgazines 
—black natk)nal organizattons with 

regk)nai and k>cat ai^SNtea 
—t>iack colleges and churches 



See page 14. 



Standard tracking procedures 

Consultatkxi with evaluatkx) 
stafttoxperts 

Secondary data tracking 
Secondary data tracking 
Secondary data tracking 



Evaluatkxi and tracking studies 
Anecdotal and observattonal 
informatton 



The Good News n)edia campaign 

The ''Quit for Good*' program for 
physicians 

Partners in Preventton Network 

Placing staff In policy-setting 
positfons 

Mall-Intercept PSA tests 
Focus group poster tests 

Pitot test of pharmacy-based smoking 
ce&satkKi program 

Pitot test of nrKXhilar TV nutrltfon 
program 

Mailed announcenoents of ''CXiit 
for Good" kit using professtonal 
societies' mailing lists 
Newsletter announcements 
TV PSA campaign 

See page 15. 



TV PSAs 

a) bounceback cards analysis 
surveys of statfons 

b) BAR public service nf)onitoring data 

c) BAR data/Inquiry data 

d) Analysis of inquiry data 



Review might indicate the need for 
improved promotfon, revised products, 
selectfon of cKfferent intermecKaries 
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Selecting Channels and Materials 




J 



With investigation and analysis 
completed and your plan in 
hand, you are ready to devel- 
op the corrvnunication tools. It is vital 
to have the right tools for the job. In 
stage 2 you1l decide what kinds of 
materials fit your budget and your 
target audience, and which channels 
will work the best. Choosing nnes- 
sages. fomnats and channels are 
interdependent steps— each must suit 
the other, and each deciskxi must fit 
the strategy statenvent. 

MtnlNjfMMsagMMHi 



Materials productfon can be a time 
consuming and costly process. 
Because it is creative and has tangible 
results, it is frequently seen as the key 
devBtopm*. ntal step lor a communica- 
tton program. However, you should 
consider whether creating new 
materials is really necessary. 




Mxi may have discovered existing 
communicatkxi materials (booklets, 
leaflets, peters, public S6nm:e 
announcements. vkJeotapes) while 
gathering data to plan your program. If 
not, kx>k now. Sources include: 

■ health departments (in your state or 
other states) 

■ university or public libraries 

■ voluntary organizations 

■ health professkxial associations 

■ community-based health promotion 
coalitkxis 

■ the ODPHP National Health 
Information Center (see page 75). 



If you find materials related to your 
health issue, deckle whether they 
might be appropriate for your program 
"as is" or witn noodificatton. ConskJer 
these issues in conjunctkxi with your 
sfrafegystaJemer* 

■ Do they offer accurate, complete 
and relevant messages? 

■ Are they appropriate for your target 
audience in format, style and read- 
ability level? 

■ Are they available and affordable? 

■ CoukJ they be modified to become 
appropriate? 

If you are considering the use of 
existing materials, you will want to 
discuss with the producer: 

■ how the messages were developed 

■ whether the materials were tested 

■ how they have been used 

■ were they effective 

& if you could receive permission to 
modify or use parts of the materials 
(especially if materials are copy- 
righted). 

Sample forms developed by OSAP 
are included in appendix C to he«p you 
conduct your own nrwiterials review. 

Mxj may want to test promising 
materials with the target audience at 
this point (see stage 3 for a descriptton 
of testing methods). If the materials 
prove to be inappropriate you will have 
valuable infomnatkxi for developing 
new materials. 

Discussk>ns about what materials 
formaf (eg., print or audtovisual) will 
best suit your program will be 
determined by: 



■ the nr^essage (eg., its coa>plexity. 
sensitivity, styla purpose) 

■ the audience (eg., will they want to 
read about the subject, or would 
they rather watch a videotape) 

■ the channels (eg., whether you will 
be most likely to reach the audience 
through a school, library, physician, 
the media, or a combinatton cI 
these), and the formats thf channels 
prefer and 

■ your budget and other available 
resources. 

Communication nrwiterials will 
probably represent a majc^ expendi- 
ture for your program. Make sure that 
you choose a format that you can 
afford in sufficient quantities to make 
your program work, and that you don't 





allot so much of your budget to materi- 
als that you can't afford distribution 
(e.g., nnalling lists, mailing services, 
postage costs), promotion and process 
evaluation. 

ChooMChanntIs 

The decision regarding the channels 
your program will use is interdepend- 
ent with the decision about materials 
format. Message delivery channels 
include: 

■ face-to-face (eg., health care profes- 
sional to patient, peers, family 
menrtoers) 

■ group delivery (eg., worksite or 
classroom) 

■ organizational (e.g., constituents of a 
professional or voluntary association) 

■ mass media (e.g., radia television, 
magazines, direct mail, billboards, 
transit cards, newspapers) 

■ community (eg., libraries, employ- 
ers, schools, malls, health fairs, local 
government agencies^ 

■ a combination of any or all of these 
(this choice is likely to work best). 

Each channel offers different benefits 
and may require different message 
design (althoi^gh not necessarily 
different messages) to fit the channel 
in length and fomiat. 

The mass nr)edia can transmit news 
quickly to a broad audience, but can- 
not alone be expected to nx)tivate 
people to change their behavior. The 
mass media are generally the public's 
primary source of information out may 
be less trusted than nx)re intimate 
sources of information, and are 
con?5trained by time, space, and 
newsworthiness. among other factors, 
to the extent they can explain complex 
intormatton properly and fully Media 
(news) may focus too much attention 
on new information or information 
affecting limited segments of the 
population; may increase the chances 
for miscommunication of complex or 
controversial news; may communicate 
incomplet: information (most crucially, 
leaving out infornr,ation explaining what 



shoukJ be done about a health 
problem). Loss of control over how the 
information is communicated may be a 
tradeK)ff for broad and rapid 
transmission. 

Whether mass media is intentionally 
selected as one channel, or whether a 
health issue appears as news, you 
should remember that the purpose of 
the mass media is to inform and enter- 
tain, not educate Therefore, if the 
complete message is too complicated, 
or simply not considered interesting 
enough for use by the media, you will 
be obligated to redesign the message 
so that it is nriore appealing to media 
professionals (media "gatet(eepers'*) 
and their perceptions about what their 
audience wants. Working with media 
professionals will help assure that 
messages are interesting as well as 
accurate, and may help you obtain 
greater exposure for your program. 

Mass media offers many opportu- 
nities for messages beyond public 
service announcements, including 
mentions in news programs, enter- 
tainment programming, public affairs, 
"magazine" and interview shows 
(including radio audience call-ins), live 
remote broadcasts, editorials (tele- 
vision, radio, newspapers, magazines) 
and health and political columns In 
newspapers and magazines. Each 
fomnat offers a particular advantage for 



communicating messages, and each 
format may reach a different audience 
mix. Mxi may decide to include a 
variety of formats and media channels, 
always choosing from among those 
nriost likely to reach your target 
audiences. 

Interpersonal channels put health 
messages in a more familiar context. 
These channels are more likely to be 
trusted and influential (physicians, 
friends and family members of the 
target audiences are usually very 
credible sources). Developing 
messages, materials and links into 
interpersonal channels may require 
time-consuming and costly develop- 
ment. Influence through interpersonal 
contacts requires familiarity with the 
message and, therefore, may more 
appropriately follow long-term expo- 
sure to and acceptance of media 
messages. 

Community channels (including 
schools, employers and community 
organizations) can reinforce and 
expand upon media messages, and 
offer instruction. Establishing links with 
community institutions and organiza- 
tions can shortcut the development of 
interpersonal routes of influence with 
the target audience Interpersonal and 
community channels can offer support 
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for action and are two-way. allowing 
discussion and clarification, encour- 
aging motivation, and reinforcing 
action. 

Using several different channels will 
increase the repetition of the informa- 
tion, increasing the chance thc*t the 
audience will be exposed to the infor- 
mation a sufficient number of times to 
absorb and remember it. Channel 
selection should be determined prior tc 
materials production since message 
format will be different for various 
channels. 

Consider these questions as you 
make decisions about channels: 

■ Which channels are most appro- 
priate for the health problem/issue 
and message? 

■ Which channels are most likely to be 
credible to and accessible by the 
target audience(s)? 

■ Which channels fit the program 
purpose (eg., inform, 'nfluence 
attitudes, change behavior)? 

■ Which and hovv many channels are 
feasible, considering your time 
schedule and budget? 

ybu may chooi^e a different mix of 
channels fa each audience to match 
what you have learned about what 
information sources the audience 
considers credible and how they are 
most likely to be reacheo. 



Selected Readings 



Burrelle's Media Directories, Burrelle's 
Special Groups Media Directory, Update 
(updated regularly), Livinyston. NJ: 
Burrelle's Media Directories. 

Cable TV Publicity Outlets, Cable TV 
Publicity Outlets— Nationwide, 
Washington, Depot. CI: Cable TV Publicity 
Outlets, updated regularly. 

Gannett Newspapers. Media Market 
Guide, New Vbrk: Gannett Newspapers, 
updated regularly. 

Papazian, E.. The Media Book. New M^rk: 
The Media Book. 1980. 

Roman, Kenneth, and Maas. Jane. How to 
Advertise, St. Martin's Press, 175 Fifth 
Avenue, New >t)rk, NY 10010. 1976. 

Simmons Market Research Bureau. 
Simmons 1985 Study of Media and 
Markets, Media Series. Vols. 1-13. New 
York: Simmons Market Research Bureau. 
Inc.. 1985. 

Storer Communications. Broadcasting 
Yearbook, Hollywood and NewMDrk: 
Broadcasting Publications Inc.. updated 
yearly. 

Traub. James, "The World According to 
Nielson: Who Watches Television and 
Why," C/?an/?e/s. January/February 1985. 
p. 26. 
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ith program planning com- 
pleted, you are ready to pro- 
duce nnessages and nnaterials 



for your progrann. This chapter ad- 
dresses: 

■ developing nnessage concepts 

■ testing nnessage concepts and draft 
nnaterials 

■ production tips 

■ content issues, including message 
appeal, spokespersons and consid- 
erations for "hard-toreach" or other 
special audiences. 

If you have located suitable materials 
for yoir program, you will be able to 
skip or simplify this process. If you are 
unsure whether the materials you have 
located are appropriate — or think that 
they may need alterations— you can 
move directly to pretesting. 

Develop and Test Message 
Concepts 

Your connmunication strategy state- 
ment and the iniormation you gathered 
about the target audiences (stage 1) 
form the basis for developing message 
concepts. These message concepts 
are your messages in "rough draft," 
and represent different ways of 
presenting the information to the target 
audiences. You may want to prepare 
two or more message concepts using 
different: 

■ spokespersons (e.g., a physician, a 
peer) 

■ appeals (e.g., humor, fear, factual) 

■ Siyles (e.g., photographs, graphs) 

■ formats (e.g., audiovisual with music, 
instructional poster) for testing with 
gatekeepers and the target audi- 
ence. 



AudtovltualMatMrlalt 

If you have decided to produce PSAs 
or other audiovisual materials, here are 
some presentation options to consider: 

demonstration — the8"<^' -(sual 
format is ideal for demonstrating the 
desired health behavior, especially if 
skills must be taught. 

testimonials—a credible presenter 
(e.g.. the Surgeon General) can lend 
credibility to your message. A rec- 
ognizable spokesperson may be 
attention-getting. The most credible 
and relevant presenter may vary for 
different audiences— and may be a 
target audience representative, an 
authority (eg., a physician) or a 
celebrity connected with the health 
issue. 

slice of life— a dramatization within 
an ''everyday" — or familiar— setting 
may help the audience associate with 
your message. A simple story may be 
easy to remember; you might choose 
to present the health problem and 
show the solution. This style may be 
both credible and memorable, but it 
may also be "corny." 




% 


f - 

V. 
















■ 




7 



an/maf/on— beyond use for young 
children, animation can be eye- 
catching for adults (older children may 
consider some animation "silly'' or 
"babyish"). You can use animation to 
demonstrate desired actions; it may 
also be a good choice for addressing 
abstract subjects (e.g., explaining 
"respite care") or sensitive subjects 
(such as AIDS), or several disparate 
target audiences (e.g., different ethnic 
groups) at once. 

humor — can be memorable, heart- 
warming and effective. However, 
humor is difficult to do well. The 
lighthearted can also be silly and a 
punchline can become stale quickly 
For some audiences or subjects, 
humor can be offensive. ("There is 
nothing funny about cancer," agreed 
one men's focus group.) It can also be 
expensive. 

emotion — can make a message 
real, and personal; It can also "turn 
off" the\ ower. Emotional approaches 
range from warm and caring to fear 
arousing and disturbing. As with 
humor, emotional appeals may be 
"high risk" production choices — these 
choices should be pretested and 
produced with ca^e. 

use of music — can lend to a mood 
you are trying to creaie* it can also 
compete with the message. 
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Print Mfltorteli 

Mdu may decide to produce a single 
booklet, or your program and budget 
may call for a range of materials, each 
for a different purpose (e.g.. a poster 
for attention, a booklet for explanation 
or teaching) or use (in a supermarket 
display a ph^ ^ician's office or a class- 
room). 

As you decide what print materials 
to produce, consider that: 

■ all messages in all the media you 
chOuse should reinforce each other 
and follow the communications 
strategy 

• no matter how creative, compelling 
or wonderful a message is. if it does 
not fit the strategy statement, objec- 
tives and identifieid audiences, throw 
it out. Don't compete for attention 
with your own campaign 

■ whatever approach or style you have 
chosen should be echoed in all 
campaign elements. In print ma- 
terials, use the same or compatible 
colors, types of illustrations and 
typefaces throughout the campaign. 
If there is a logo or theme, use it in 
all print and audiovisual materials 
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■ use illustrations to gain attention, aid 
understanding and recall; make sure 
they reinforce, and don't compete 
with your message. Use captions, 
headlines and summary statements 
for additional reinforcement. 

As you consider a particular tone or 
style for either audiovisual or print 
materials, nnake sure that you have 
access to the expertise and budget to 
suit your choice. Think about whether 
the style will enhance or compete with 
your messaga Simplicity may offer the 
greatest chance of success. 

Production Valim 

No matter which approach you 
choose, high quality production is 
necessary to make the message work. 
If you feel you have to skimp on pro- 
duction, choose a simpler way of 
presenting the message. Producing 
poor quality materials wastes funds 
and can damage your program's— and 
your own — credibility. 

Nood for Aiidl«nco Totting 

Even If you think you've chosen the 
presentation style most suited to the 
message and audience you should 
pretest it to be sure. Check each 
concept to make sure that it complies 
with your communications strategy 
and objectives, "testing alternative 
concepts with the target audience 
may: 

■ help identify which has the strongest 
appeal and potential for effect 

■ Identify new concepts 

■ identify confusing terms or concepts 

■ identify language used by the target 
audience 

■ help eliminate weaker concepts and 
save production costs. 




Focus groups are most commonly 
used for testing at this stage because 
they permit open and extended discus* 
sion about concepts and ideas. You 
may show rough illustrations or a tele- 
vision "storyt)oard" (frame-by-frame 
illustrated description) to the group, or 
just discuss the messages and presen- 
tation style with them and ask them to 
visualize the product. A description of 
focus groups and alternative test 
methods begins on page 39. 

Dtvtlop Draft Materials 

Based on findings from testing mes- 
sage concepts, you will want to refine 
the most promising approach and 
produce materials in draft or "rough" 
form. A graphic designer or audio- 
visual producer v;ill frequently provide 
a facsimile version of a poster or 
pamphlet, or a storyboard of a 
television PSA for your review and 
approval. Materials in — or close to— 
this format should be tested at this 
stage to: 

■ assess comprehension 

■ identify strong and weak points 

■ determine personal relevance 

■ gauge confusing, sensitive or 
controversial elements. 
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Testing at the concept stage helps to 
choose the strongest fronn among 
more than one potential approach to 
presenting your message. Now, you 
have refined the strongest concept 
into the language, style and format you 
want to use Additional testing is de- 
signed to strengthen the approach you 
have chosen— before your production 
funds are spent. 

Because materials production is 
one of the most costly steps in pro- 
gram development, it makes sense to 
test before you invest in final produc- 
tion. Draft materials should resemble 
the final product as closely as 
possible, but without expensive pro- 
duction. For example, radio announce- 
ents may be produced in a nonsti"^io 
setting and with nonprofessional 
voices for testing. A television an- 
ouncement may be tested wi:h a story- 
board, or with an animatic (videotaped 
sketches with voice over). A booklet 
may be p epared using good quality 
word processing copy, rough illus- 
trations, and a copy machine. Similarly, 
posters, print ads and flyers can be 
produced in rough form for pretesting. 



Pretest graphics with the target 
audience. People interpret graphics in 
different ways, just as they do the 
written word. If your graphics style or 
illustrations depart from what 
"gatekeepers'* (e.g., PSA directors, 
physicians, teachers) exoect, test with 
them, or ask them to review, as well. 
Use favorable responses from the 
target audience as a '^selling point" 
with gatekeepers. 




PratMting-What K Can and 
Cannot Do 



Pretesting draft materials is a type of 
formative evaluation used to help 
ensure that communications materials 
will work. Pretesting is used to answer 
questions about whether materials are: 

■ understandable 

■ relevant 

■ attention-getting and memorable 

■ attractive 

■ credible 

m acceptable to the target audience. 

These are factors that can make the 
difference in whether materials v\/ork or 
don't work with a particular group: they 
also involve value judgments on the 
part of th<rf respondents and your 
interpretation of what they mean. Most 
pretesting involves a few persons 
chosen as representative of intended 
target audiences, and not a statistical- 
ly valid sample (in number or selection 
method). That is, pretesting is general- 
ly considered ''qualitative research" — 
research which can be interpreted 
somewhat loosely to provide clues 
about audience acceptance and 
direction regarding materials pro- 
duction and usa 




A variety of procedures may be 
used to test messages and materials 
The best methods for a particular 
program depend upon the nature of 
the materials, the target audience and 
the amount of time and resources 
available for pretesting. There is no 
formula for selecting a pretest 
methodology, nor is there a "perfect" 
method for pretesting. Methods should 
be selected and shaped to fit each 
pretesting requirement, considering 
the objectives of and resources 
available for each project. 

Included here are descriptions of 
some frequently used methods of 
pretesting health concepts, messages 
and materials. In addition, sample 
questionnaires or other forms that 
have been developed for some of 
these methods are included in 
appendix C, for you to adapt. Each 
method carries with it benefits and 
limitations. Sometimes using severa/ 
methods in combination will help 
overcome the limitations of individual 
procedures For example, focus group 
interviews may be used to identify 
issues and concerns relative to a 
particular audience, followed by 
individual interviews to discuss 
identified concerns in greater deptn 
Readability testing should be used as 
a first step in pretesting draft manu- 
scripts, followed by individual question 
naires or interviews regarding materi- 
als with target audience respondents 
Central location interviews or theater 
testing of messages for television or 
radio permit contact with larger 
numbiers of target audience respond- 
ents—especially useful prior to final 
production of materials Following the 
descriptions of pretesting methods is a 
discussion of how to choose the most 
suitable method for a particular 
situation 




Given the qualitative nature of most 
pretesting research, it is important to 
recognize its limitations: 

■ Pretesting cannot absolutely predict 
or guarantee learning, persuasion, 
behavior change or other measures 
of communication effectiveness. 

■ Pretesting in health communication 
is seldom designed to quantitatively 
measure small differences among 
large samples; it '3 not statistically 
preci >e It will not reveal that booklet 
A is 2.5 percent better than booklet 
B. (Presumably, pretests of such 
precision could be applied, but the 
cost of obtaining such data woukJ be 
high, and the findings may be no 
more useful than the diagnostic 
information fr'^m more affordable 
approaches.) 

■ Pretesting is not a substitute for 
experienced judgment. Rather, it can 
provide additional information from 
which you can make sound 
decisions. 

It IS important to avoid misuse of 
pretest results. Perhaps the most 
common error is to overgeneralize. 
Qualitative, diagnostic pretest methods 
should not be used to estimate broad- 
scale results. K 5 of the 10 respond- 
ents In a focus group interview do not 
understand portions of a pamphlet, it 
dof ? 'tOt necessarily mean that 50 
percent of the total target population 
will be confused. The lack of under- 
standing arTX)ng those pretest re- 
spondents suggests, however, that the 
pamphlet may need to be revised to 
Improve comprehension. In sum, 
pretesting is indicative, not predictive. 

Another problem that arises in 
health cammunication pretesting 
concerns interpretation of respondent 
reactions to a sensitive or emotional 
subject such as breast cancer or 
AIDS. Respondents may become 
unusually rational when reacting to 
such pretest materials, and cover up 
their true concerns, feelings and 
behavior. As a result, the pretester 
must examine and interpret responses 
carefully. 



Pretesting offers both the opportu- 
nity and the temptation to structure the 
test and interpret ths results to support 
or justify a preconceived point of vievA 
It is natural to want your favorite con- 
cepts or messages to test well, but 
there is no need to test unless you are 
willing to consider the results ob- 
jectively. 

One final point: pretesting does not 
guarantee success. Good planning 
and sound pretesting can be negated 
by mistakes in final production. The 
message in a television PSA on cancer 
treatment, for instance, may pretest 
well, but then be flawed by an execu- 
tion that uses an actress who seems 
too happy to be awaiting the results of 
a biopsy report. Similarly leaflet copy 
that pretests well may be rendered 
ineffective by a poor layout, hard-to- 
read type, and inappropriate 
illustrations. 

Pretesting Methods 

The most frequently used pretesting 
methods are described below. These 
include: 

■ self-administered questionnaires 

■ central location intercept interviews 

■ theater testing 

■ focus group interviews 

■ readability testing 

■ gatekeeper review. 

Following these descriptions there is 
a summary chart on page 47 to help 
you compare the advantages and dis- 
advantages of each method. 

1. S#lf-«dmlnitt«rtd Qu«stlonnairts 

Self-administered questionnaires: 

■ enable program planners to elicit 
detailed information from respond- 
ents who may not be accessible for 
personal interviews (e.g., doctors, 
teachers or residents of rural areas) 

■ allow respondents to maintain their 
anonymity and reconsider their 
responses 

■ do not require interviewer time and 
can be done relatively inexpensively 
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■ can be answered by many respond- 
ents at once 

■ require time to locate respondents 
and secure their cooperation 

■ require follow up to increase re- 
sponse rates if mailed. 

Self-administered questionnaires 
also can be: 

■ mailed to respondents along with the 
pretest materials 

■ distributed to respondents gathered 
at a central location 

■ used where personal interviews are 
not feasible 

■ an inexpensive pretesting technique 
for agencies with minimal resources. 

A self-administered questionnaire 
should be designed (see sample in 
appendix C), then pilot tested with 5 to 
10 respondents. Usually, question- 
naires and pretest materials are dis- 
tributed to respondents whose partici- 
pation is sought in advance, but they 
also may be mailed to potential 
respondents without advance noti- 
fication Respondents are asked to 
review the materials on their own, to 
complete the questionnaire and then 
to return it within a specified time. 

The questionnaire should be rela- 
tively short and clear or respondents 
may not complete it. Clear, concise 
instructions to the respondent are 
important because there is no 
interviewer to offer clarification. Open- 
ended questions may be used to 
assess comprehension and overall 
reactions to materials and close-ended 
questions to assess such factors as 
personal relevance and believability of 
the material. Measures of attention or 
recall may not be reliable when used 
with this technique since respondents 
may refer back to the material. 

Resources are invested primarily in 
questionnaire development and anal- 
ysis of results. The latter expense can 
be kept to a minimum by using many 
close-ended questions. 




Self-administered questionnaires 
have certain disadvantages: 

■ The primary problem with this 
technique is the possibility of a low 
response rate. 

• It is important to over-recruit re- 
spondents and recontact respond- 
ents to encourage them to return 
their questionnaires to ensure a 
sufficient number of returns. 

■ The data collection may take longer 
than with other methods (e.g., 
central location intercept interviews) 
because of delays in responses, 
especially if the questionnaires are 
mailed. 

■ The type of respondents who return 
the questionnaires may be different 
from those who do not respond, and 
this approach cannot be used with 
respondents who have reading and 
writing limitations. Hence, a certain 
degree of bias may be introduced 
and results should be interpreted 
with this In mind. (Contacting those 
who did not respond by telephone 
will permit a comparison of 
respondent/nonrespondent 
answers.) 



2. Central lAcatkm intwctpt intcnritws 

Central location intercept interviews in- 
volve stationing interviewers at a point 
frequented by individuals from the 
target audience and asking them to 
participate in the pretest. There are 
two adv^.titages to this: 

■ a high traffic area (e.g.. a shopping 
mall, hospital waiting area or school 
yard) can yield a number of inter- 
views in a reasonably short t'T'e 

■ a central location for hard-to-reaci 
target audiences can be a cost- 
effective means of gathering data. 

A typical central location interview 
begins with the intercept. Potential 
respondents are stopped and asked 
whether they will participate. Then, 
specific screening questions are asked 
to see whether they fit the criteria of 
the target audience. If so, they are 
taken to the interviewing station — a 
quiet spot at a shopping mall or other 
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site — are shown the pretest materials, 
and asked questions. The questions 
nnay help assess: 

■ comprehension 

■ individual reaction 

■ personal relevance 

■ credibility 

■ recall (if test situation includes 
exposure to the materials prior to the 
interview). 

Although the respondents inter- 
cepted through central location 
interviews may not be statistically 
representative of the target population, 
the sample is usually larger than those 
used in focus groups or individual in- 
depth interviews. 

Unlike focus groups or in-depth 
interviews, the questionnaire used in 
central location intercept pretesting is 
highly structured and contains pri- 
marily multiple choice or close-ended 
questions to permit quick response. 
Open-ended questions, Vv'hich allow 
"free flowing" answers, should be kept 
to a minimum because they take too 
much time for the respondent to 
answer and for the interviewer to 
record responses. The questionnaire, 
as in any type of research, should be 
pilot tested before it is used in the 
field. A sampL*^ questionnaire is 
included in appendix C. 

A number of market research 
connpanies throughout the country 
conduct central location intercept 
interviews in shopping malls. Clinic 
waiting rooms, churches, Social 
Security offices, schools, worksites or 
other locations frequented by individu- 
als representative of the target audi- 
ence also can be used for this 
purpose It is advisable to obtain 
clearances or permission to set up 
interviewing stations in these locations 
well in advance. 

Posters can be tested in the kind 
of setting (e.g., a clinic waiting room or 
schoolroom) where they will be used. 
Posters should be mounted on a wall 
along with other materials—just as 
they are expected to be used— where 



the target audience passes, gathers or 
waits. Selecting respondents from 
among those who have been "ex- 
posed" to the poster in its "natural 
setting" prior to the interview, and then 
moving to a nearby but separate 
location to ask questions will permit an 
assessment of factors such as com- 
prehension and personal relevance, 
and also whether: 

■ the material attracts attention 

■ the lespondent can recall the 
material when exposed to it in a 
"natural" setting 

The major advantage of the central 
location intercept approach is its cost- 
effectiveness for interviewing large 
numbers of respondents in a short 
amount of time. Because these 
interviews are intended to provide 
guidance ("qualitative" information), 
the size of the sample should only be 
large enough to give you answers to 
your oretest questions. If you have 
interviewed 50 respondents and most 
of them feel similarly about your 
materials, you are probably ready to 
stop. If, however, there are substantial 
disagreements or differences between 
respondents, or their responses have 
raised new questions, additional 



interviews should be conducted until 
you are satisfied that you have clear 
direction from the respondents. You 
may decide to revise (and perhaps test 
again) after fewer interviews if it is 
clear that changes are needed. 

Designing a central location inter- 
cept pretest can be relatively easy. A 
few simple questions ("Do you 
smoke?" "How old are you?" "Do you 
have teenaged children?") can identify 
respondents typical of the target audi- 
ence quickly at the point of intercept 
Questions to assess comprehension 
and target audience perceptions of the 
pretest materials form the core of the 
questionnaire (see appendix C). A few 
additional questions, tailored to the 
specific item or items being tested 
("Do you prefer this picture— or this 
one?"), also may be constructed to 
meet program planners' particular 
needs. The interview should be no 
longer than 15-20 minutes. If it must 
be longer, you may need to design 
special incentives to convince the 
respondent to continue the interview 
(e.g., a small fee or gift, or a plea 
regarding the importance of the 
subject and their opinions). 
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Central location intercept interviews 
should not be used if respondents 
must be interviewed in depth or on 
emotional or very sensitive subjects. 
The intercept approach also may not 
be suitable if respondents are likely to 
be skeptical or resistant to being 
interviewed on the spot (eg., 
connmuters anxious to return home). 
Although it is time-consuming to set up 
prearranged appointments, they may 
actually save time if respondents may 
not be willing to cooperate in a central 
location. 

3. ThtitorTMting 

"Theater" tests are so-called because 
they gather a large group of respond- 
ents to react, usualiy to audio or 
audiovisual materials, into a room (or 
"theater" -style setting) at once. 
Commercial services conduct theater- 
style tests for advertising agencies; 
this technique can be adopted for 
health messages. In commercial 
theater testing, approximately 300 
respondents are recruited by tele- 
phone to a central location, such as a 
hotel. Respondents are asked to watch 
a "pilot" television program to judge 
whether it should be aired. Commer- 
cials are included in the program; 
Sonne are control (constant) spots, 
others are being tested. At the conclu- 
sion of the program, respondents are 
asked whether they recalled any 
connmercials (or PSAs), and then 
asked questions regarding content and 
personal relevance. A similar se- 
quence can be used to test radio 
connmercials. 

Theater testing quickly gathers a 
large numt)erof responses. Unlike 
Sonne other pretest methods, the 
materials feeing tested are imbedded 
within a program, with commercials, to 
simulate a natural viewing situation. 
This permits the assessment of how 
likely the audience is to pay attention 
to and rennember the message. 

Because commercial testing 
services are costly, a guide to con- 
ducting your own theater-style tests is 
included in appendix D. You can 



choose a setting where the target 
audience gathers and where they can 
assemble in a large group (e.g., a 
senior citizen center, a school 
auditorium) to conduct your own 
theater-style test. 

4. Focus Qroup bittnritws 

Focus group interviews are a form of 
qualitative research adapted by market 
researchers from group therapy. A^so 
called exploratory group sessions, they 
are used to obtain insights into target 
audience perceptions. beMs and 
language. Focus group interviews are 
conducted with a group of about 8 to 
10 people. Using a discussion outline, 
a moderator keeps the session on 
track while allowing respondents to 
talk freely and spontaneously. As new 
topics related to the outline emerge, 
the moderator probes further to gain 
useful insights. 

Focus group interviews are es- 
pecially useful in the concept develop- 
ment stage of the communication 
process. They provide insights into 
target audience beliefs on a health 
issue, allov. program planners to 
explore perceptions of message 
concepts, and help trigger the creative 



thinking of communication profession- 
als. The group discussion stimulates 
respondents to talk freely, providing 
valuable clues for developing materials 
in the consumers' own language; and 
suggestions for changes or new 
directions. 
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In the planning stages of program 
development focus groups can be 
used to develop the hypotheses (or 
broad study issues) for larger quantita- 
tive studies. Focus groups also can 
help determine public perceptions, 
misconceptions and attitudes before a 
questionnaire is developed and the 
field research is conducted. 




Focus groups also can be used to 
supplement quantitative research. 
Market researchers originally devel- 
oped this technique to explore in 
greater depth the data from large scale 
consumer surveys. Obtaining in-depth 
information from individuals typical of 
the target audience can provide 
insights into what the statistical data 
mean, or why individuals respond in 
certain ways. 

As with all respondents, those 
selected for focus groups should be 
typical of the intended target audience 
Various subgroups within the target 
audience may be represented in sep- 
arate group discussions, especially 
when discussing sensitive or emotion- 
al subjects, to segregate respondents 
by age, sex, race or whatever other 
variable is likely to hinder freedom of 
expression. Teenage girls are less 
likely to be inhibited in discussing 
sexual activity, for instance, if their 
parents, or teenage boys, are not in 
the group. Respondents are recruited 



1 to 3 weeks in advance of the inter- 
view sessions, usually by telephone. 
They may be recruited using the 
telephone directory, and interviewed by 
phone to determine if they qualify for 
the group. Or, they may be recruited 
from among members of a relevant 
organization, place of employment or 
other source. Recruiting respondents 
''at random" is not required because 
the results from focus group research 
are not intended to be statistically 
representative. 

There are several important criteria 
for conducting effective group inter- 
views. Ideally respondents should not 
know the specific subject of the 
sessions in advance, and they should 
not know each other. Knowing the 
subject may result in respondents 
formulating ideas in advance and not 
talking spontaneously about the topic 
during the session. Knowing other 
respondents may inhibit individuals 
from talking freely. Finally, all respond- 
ents should be relative "newcomers" 
to focus group interviews. This permits 
more spontaneity in reactions and 
eliminates the problem of "profession- 
al" respondents who may lead or 
monopolize the discussion. For the 
same reasons you may want to 
exclude health professionals and 
market researchers from focus groups 

There is no firm rule about the 
number of focus groups that should be 
conducted. The number of groups 
depends upon program needs and 
resources. If target audience percep- 
tions appear to be comparable after a 
few focus groups (you'll need at least 
two groups to make this decision), you 
may not find out any more by con- 
vening additional sessions. If per- 
ceptions vary, and the direction for 
message development is unclear, 
additional groups may be beneficial. In 
this case, rev.sions in the discussion 
outline after several groups can help 
clarify unresolved issues in the 
additional groups. 




An experienced, capable moderator, 
who can skillfully handle the group 
process, should be used. The moder- 
ator does not need to be an expert in 
the subject matter being discussed; 
rather, a good moderator builds rap- 
port and trust and should probe 
respondents without reacting to, or 
influencing, their opinions. The 
moderator must be able to lead the 
discussion, and not be led by the 
group. The moderator must emphasize 
that there are no right or wrong 
answers to questions posed. A good 
moderator understands the process of 
eliciting comments, keeps the dis- 
cussion on track, and makes it clear 
that he or she is not an expert on the 
subject. You will need to ehearse with 
the moderator to point out any topics 
or concerns you want emphasized, or 
discussed in more depth. 

As noted earlier, the results of focus 
group interviews should be interpreted 
carefully. It is useful for an unseen 
observer (e.g., behind a one-way 
mirror) to take notes as well as to tape 
record or videotape the session for 
later review. In interpreting the findings 
from group interviews, you should look 
for trends and patterns in target audi- 
et ice perceptions rather than just a "he 
said ... she said" kind of analysis. 
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Group discussion should not be 
used when individual responses or 
quantitative information are needed. 
For example, when assessing the final 
copy for a booklet, it is more important 
to gather Individual ra\her than group 
reactions to indicate the individual's 
actual comprehension, perceptions 
and potential use. However, self- 
administered questionnaires can be 
completed by each participant prior to 
beginning a group discussion to 
combine individual and group 
reactbns. 

5. RMdabillty Totting 

"Readability testing" simply predicts 
the approximate educational level a 
person must have in order to under- 
stand written materials. Health infor- 
mation materials such as pamphlets, 
flyers, posters and magazine articles 
are designed for distinct target groups: 
a readability test will indicate if a 
printed piece is written at a level most 
of the audience can understand. As- 
sessing the readability of a pamphlet 
or another printed message will not 
guarantee its effectiveness, and is by 
no means an absolute indicator of 
success. 

Readability formulae use counts of 
language variables such as word and 
sentence length. The formulae have 
been devised statistically to predict 
readability scores. Generally speaking, 
the reading level required to under- 
stand a given pamphlet wiii be higher 
when its sentences are long and/or 
when a large number of polysyllabic 
words is found within the text. 

It is important to note that readabil- 
ity formulae measure only the structur- 
al difficulty (i.e, vocabulary, sentence 
structure and word density) of written 
text. They do not measure other 
factors related to how "readable" a 
certain text is, such as sentence 
"flow," conceptual difficulty organiza- 
tion of material, the influence of format 
or design of materials on comprehen- 
sion, accuracy or credibility Readabili* 
ty tests are conducted by program 



staff and do not include participation 
by the audience for whom the materi- 
als are being produced. Consequently 
readability testing supplements but 
does not supplant the need to pretest 
with the target audience. 

Despite its limitations, readability 
testing is useful because it: 

■ is quick 

■ is virtually without cost 

■ provides a tangible measure 

■ reminds the writer to choose words 
and terms carefully 

Based on a review of the advan- 
tages, disadvantages and predictive 
validity of 12 selected readability 
formulae, the NCI Office of Cancer 



Communications chose the SMOG 
grading formula for testing the 
readability levels of its public and 
patient education materials. SMOG 
was chosen because it is both simple 
to use and accurate. Complete 
instructions for using the SMOG 
readability test to print materials are 
included in appendix B. 

Health and medical subjects often 
include many polysyllabic words and 
complex terms; readability formulae 
have not been designed to take into 
account the special terminology used 
in describing health subjects. In some 
cases, extensive use of multisyllabic 
words known to be understandable to 
a particular audience (e.g.. "cigarette") 
may lead to an unwarrantedly high 
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readability score. Therefore, as with all 
pretesting, readability test results 
should be used as indicative and not 
predictive of problems or success. 

6« QattkMp^r Rtvitw 

Often, public and patient education 
nnaterials are routed to their intended 
target audiences through health pro- 
fessionals or other intermediaries such 
as organizations that can communi- 
cate for you to their members. These 
intermediaries act as "gatekeepers," 
controlling the distribution channels for 
reaching your target audiences. Their 
approval or disapproval of materials 
may be a critical factor in a program's 
success. If they do not like a poster or 
a booklet, or do not believe it to be 
credible or scientifically accurate, it 
may never reach the intended audi- 
ence. Also, because they may be in 
closer touch w«th the target audience 
than you are, they may provide good 
advice about whether the audience 
will accept the materials. 

Although not a pretesting technique 
in the strictest '^.ense of the term, 
gateteeper :eview of rough materials 
is important and should be considered 
part of the formative evaluation pro- 
cess. It IS not a substitute for pretest- 
ing materials with target audience 
representatives. Neither is it a 
substitute for obtaining clearances or 
expert review for technical accuracy; 
these should be completed before 



pretesting is undertaken. Sometimes 
telling the gatekeeper that technical 
experts have reviewed the material for 
accuracy may provide reassurance 
and hasten gatekeeper approval. 

Gatekeeper reviews may be 
conducted simultaneously with target 
audience pretesting so that data from 
both groups can be gathered, analyzed 
and synthesized to provide direction 
for revising materials. A short, self- 
administered questionnaire may be 
directed to individuals representative 
of the gatekeeper population. A 
sample is included in appendix C. 
Questions may include overall reac- 
tions to the materials and assess- 
ments of whether the information is 
appropriate and useful. 

In other cases, there maybe no 
formal questionnaire, but rather a 
telephone or peisonal conversation or 
meeting held to review and comment 
on (or approve) materials. If there is no 
questionnaire, you should consider in 
advance what kind of questions you 
want to ask in the meeting or interview 
and whether or not you need formal 
approval of the materials. A discussion 
With gatekeepers (e.g , a television PSA 
director, the executive director of a 
medical society) at this point can also 
be used to introduce your program and 
solicit their involvement in a variety oi 
ways beyond materials development. 
(See Stage 4: Implementing Your 
Program.) 



Determine What and How 
Much to Test 

Qualitative research should be 
conducted in the early stages of 
program development before full funds 
have been committed to materials 
production and messages can be 
changed if necessary. As noted earlier, 
testing can be useful at the concept 
development stage, once audiences 
and communication strategies have 
been determined, and prior to mes- 
sage development. Exploration with 
the target audience at this stage, most 
frequently through focus group 
discussions, can help determine 
appropriate message appeals (e.g., 
fear arousing vs. factual), spokes- 
pi^rson (e.g., a scientist, public official 
or member of the target audience) and 
appropriate language (determined by 
listening to the group discussion). 

Testing of drafted materials prior to 
final production permits identification 
of flaws prior to the expenditure of 
funds for final production, and 
especially prior to the use of materials 
with target audiences. 

Completed information materials are 
sometimes tested prior to beginning a 
new phase of a pre-existing program 

A combination of methods can be 
used to assess an audience's compre- 
hension, the message's believability 
personal relevance, acceptability and 
other strong and weak points. Methods 
should be selected to suit the purpose 
of the testing, the sensitivity of the 
subject and the resources available for 
testing. Adequate investigation is 
especially important when developing 
sensitive or potentially frightening 
messages, presenting complex, new 
information or designing a new 
program. In these cases, pretesting 
can reveal potential problems, but 
must be carefully structured, con- 
ducted and analyzed. 
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Qualitative research responses 
cannot be considered representative of 
the public or projectable to the popu- 
lation as a v\^hole. If projectable data 
are required, more formal methodolo- 
gies should be used. Hovy^ever. for 
most ' retesting purposes, qualitative 
methods may be more valuable 
because they provide insights into 
thinking and reasons for attitudes or 
misunderstandings that are vital to 
help ''efine nnessages and materials. 

When deciding v^hen, v^^hether and 
hov\/ nnuch you should use pretest 
methods in developing your program, 
consider: 
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■ Hovy^ much do you knovy^ about the 
target audience? 

■ Hovy^ much do you knovy^ about them 
in relation to your health problem or 
issue? 

■ Is your issue or program new, 
controversial, sensitive or complex? 

■ Have you conducted related 
research that can t^e applied to this 
topic? 

■ Can you afford to make a mistake 
W\\h a particular message or 
audience? 

Plan and Conduct Pratetts 

The level of effort and staff resources 
required v\/ill vary considerably from 
one pretest to the next. Most pretest- 
ing is conducted vy^ith small samples 
consisting of respondents vy/ho are 
typical of the target audience and vy/ho 
are easily accessible. These results, 
combined vy^ith your professional 
judgment, provide important direction 
for improving messages and materials. 

This section provides practical 
suggestions for hovy^ to plan and 
implement pretests. These sug- 
gestions should help you reduce the 
time and costs involved, vy/hether or 
not commercial research firms are 
hired to supply field vy/ork and tabula- 
tion. The cost estimates in the chart 
on page 50 are for direct costs only; 
not included are staff time to provide 
direction or other support you vy^ould 
provide to the firm conducting the test. 
In some cases, you may reduce these 
costs by conducting pretests on your 
ovy^n, vy^ith the help of an expert. Some 
market researchers vy^ill tell you that 
bad research is vy^orse than no re- 
search, and you must use profession- 
als; others say that vy/ith proper in- 
struction, you can do some testing on 
your ovy/n. Both points of vievy/ are 
valid: venture on your ovy/n vy/ith care. 

Dttigning Um Qu«ttlonnair« 

As in the planning stage of progr m 
development, a first step in planning a 
pretest is to formulate the research 
objectives. These objectives should be 
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stated specifically to provide a clear 
understanding of vy/hat you vy^ant to 
learn. Measures of attention, compre- 
hension, believability and personal 
relevance are key. Other specific 
questions to identify strengths and 
vy/eaknesses in rough messages and 
materials also should be developed 
based on the pretest objectives. 
Questions should not be asked just to 
satisfy someone's curiosity. 

There are several vy^ays to keep 
pretesting costs dovy^n: 

■ keep the questionnaire short and to 
the point 

■ try to use as many close-ended or 
multiple choice questions as 
possible for easy tabulation and 
analysis 

■ try to develop codes for quantifying 
responses in advance when open- 
ended questions are necessary 

■ vy/henever possible, borrovy^ questions 
from other pretesting research. 

Sample questionnaires are included in 
appendix C as one resource. 

Rtcruiting RMpondtntt 

If your budget does not allovy^ you to 
hire a market research firm to recruit, 
you can recruit respondents yourself. 
A small donation may encourage 
members of local church, school, civic 
social organizations to participate in a 
pretest. 

An incentive is often used to help 
ensure that respondents participate in 
a pretest. Small amounts of money 
($5-$30), gifts, movie passes or a free 
dinner may be offered as an incentive 
to participants. Another vy^ay to ensure 
sufficient participation is to recruit 
more people than are actually needed. 
Often respondents vy^ho agree to parti- 
cipate do not shovy^ up. If all partici- 
pants do shovy^ up, they should be 
included in the pretest, or the "extra" 
respondents should be informed that 
too many respondents are present, 
given the agreed-upon incentive, 
thanked, and asked to leave. 
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Other ways to increase participation 
include: 

■ scheduling the pretest at a time that 
is most convenient for respondents 
(eg., at lunch or after work) 

■ choosing a sbf<^ and convenient site 

■ providing transpOk*ation 

■ arranging for child care during the 
time of the pretest, if necessary. 

Recruiting patients or their families 
must be given special consideration. 
Clinics, hospitals or local HMOs can 
be contacted for help, and adequate 
plans should be made to ensure that 
the respondents are not inconveni- 
enced. Human subjects* clearance 
may be needed before proceeding 
Cooperation with the medical staff and 
a concern for the physical and 
emotional status of the patient and 
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family (especially If the patient needs a 
family member's assistance to attend) 
must be considered in planning the 
pretest. 

Identifying Intervltwert 

Trained interviewers should be used 
whenever possible. For focus group 
and in depth interviews, this is essen- 
tial. If your agency has no experience 
in focus group studies, you might 
consider hiring a good, experienced 
moderator, observing and taping the 
sessions and using them as training to 
develop in-house skills. Local adver- 
tising agencies may be of assistance 
in identifying a good moderator. 
Continuing education courses in 
interpersonal communication or group 
interaction also may be useful for staff 
training or identifying potential 
interviewers. 



For conducting central location 
interviews, university and college 
departments of marketing, communi- 
cations or health education might be 
able to provide interviewer training and 
student interviewers. Pretesting a 
poster or an advertisement is an 
excellent "real world'* project for a 
faculty member to adopt as a class 
project. Students in these departments 
are being trained in research methods, 
and pretesting can give them a 
chance to develop their skills. 

Pretesting facilities should be quiet 
and comfortable. Meeting rooms at 
churches, office buildings or other 
institutions can be used for conducting 
focus group or individual in-depth 
interviews. If an observation room with 
a one-way mirror is not available, you 
may allow staff to listen by hooking up 
speakers in a room nearby or by 
audiotaping or videotaping the session. 

OtttbtgHelp 

Many resources exist for obtaining 
professional assistance in pretesting. 
As mentioned in the previous section, 
the faculty at university departments of 
marketing, communications, health 
education, psychology or sociology 
can be helpful in designing and con- 
ducting pretests. Marketing research 
firms that specialize in respondent 
recruitment, interviewing, tabulation 
and other services may have facilities 
for conducting group sessions and 
other techniques. The American 
Marketing Association's Marketing 
Services Guide lists suppliers and 
services geographically throughout the 
United States. Also, advertising clubs 
(many affiliated with the Arnofican 
Advertising Federation), and chapters 
of the Public Relations Society of 
Annerica may undertake public service 
projects at no charge to nonprofit 
organizations. Other sources include 
the Marketing Research Association, 
and the Association of Public Opinion 
Researchers. 
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One caution: individuals trained in 
connmercial testing may not be 
completely aware of all the nuances 
and subtleties involved in health 
communication. They will be able to 
draw on their commercial experience 
for selecting the appropriate pretest 
nf)ethodology. However, there are other 
factors such as the wording and 
interpretation of questions and results 
that are influenced by the complexities 
of health infomr^tlon. Tb'j old adage 
that managers shc*jld Know enough 
about each facet of their business to 
manage their experts holds true for 
pretesting. >bu should be prepared to 
supervise and guide your consultants. 

SunMiMfy 

lb yield useful results, a pretest should 
be planned carefully. Ample time 
should be allowed for: 

■ contracting with research fimns (if 
necessary) 

■ arranging for the required facilities 
(1-2 weeks) 

■ developing and testing the question- 
naire (2-3 weeks) 

■ recnjiting interviewers and respond- 
ents (2-4 weeks) 

■ gathering the data (1-2 weeks) 

■ analyzing the results (1 week) 

■ making the appropriate alterations in 
messages or materials 

■ pretesting again, if needed. 

And adequate pretesting should 
include: 

■ carefully defining the target 
audience 

■ recruiting from that audience 

■ conskJerIng tests with "gate 
keepers** or intermediaries 

■ defining the purpose of materials 
prior to designing questionnaire 



■ locating a trained interviewer and 
interpreter for some tests 

■ carefully assessing results 

■ considering using a •'mix" of 
methods to tailor your pretesting to 
your needs. 

Without adequate planning, pre- 
testing may not serve its intended 
purpose — to improve your messages 
and materials. Instead, it could 
become expense research that is of 
little or no use. 
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Prepare to introduce Program 

At this point a number of actions occur 
simultaneously in order to get ready to 
introduce the program. Before it be* 
gins, materials must be available in 
sufficient quantities, program "kickoff" 
and promotion plans in place, "gate- 
keepers" representing channels of 
implementation briefed and progress 
tracking nneasures developed. For 
example, if you plan to: 

■ use the mass media to introduce the 
issue and your nev^ program (e.g., at 
a press conference) 




■ work with voluntary organizations to 
provide followup (e.g., preservations 
at their meetings) and 

■ ask the interested public to seek 
additional information from their 
physicians. 

Before you "kickoff" your program, 
consider: 

■ Do you have a list of all the relevant 
media outlets to be contacted? 

■ Does every organization that should 
be involved knov^ about your pro- 
gram? 

■ Have you prepared your staff and 
others to respond to inquiries? 

■ Do you have sufficient materials to 
start the program (e.g., PSAs and 
press kits) and respond (e.g„ leaflets 
for the public)? 

■ Are the materials in place (e.g., in 
television stations, physicians' 
offices, schools or supermarkets)? 



■ Are professionals in the community 
aware of the new program and 
prepared to respond if asked (ag., 
counsel their patients)? 

■ Do you have mechanisms in place 
to track progress (e.g., number and 
nature of inquiries) and identify po- 
tential problems (e.q . ••^c-Hicient 
supplies of materials)? 

Your program implementation 
should indicate how and when re- 
sources wil! be needed, when specific 
events will occur and at what points 
you will assess your efforts. 

The importance of Thickfng 
Progross 

Once your program is under way, you 
may not be able to anticipate all 
contingencies that may arise, but you 
can plan ways to identify potential 
problems. You should build a moni- 
toring system into your program to 
help you identify any problems, flaws 
or oversights regarding materials, 
implementation strategies or channel 
selection before they become major 
impediments to success. 

Often, problems are quickly cor- 
rectable if you can identify them, but 
can cause harm if you don't. For 
example, if you ask the public to call 
for more information, you should 
provide a mechanism (e.g., a simple 
response form) for telephone operators 
to record questions asked and 
answers given. A frequent review of 
responses will identify whether incor- 
rect or inadequate information is being 
given, any new information required to 
respond, and inquiry patterns. 

Frequently, program implementation 
takes longer than you might expect — 
materials may be delayed at the 
printer, a major news story may pre- 
empt your publicity or a new priority 
may delay community participation. A 
periodic review of planned tasks and 
time schedule will help you alter any 
plans that might be affected by unex- 
pected events or delays. There is 
nothing wrong with altering your plans 



to fit the situation -keeping in mind 
what you are trying to achieve. In fact, 
you may risk damaging your program 
if you aren't willing to be flexible and 
alter specific activities when needed. 

Process evaluation — tracking how 
and how well your program is working 
— can provide tangible evidence of 
program progress, often useful to 
provide encouragement and reward to 
participants and evidence of success 
to your own agency. It can also assure 
that the program is working the way in 
which you planned — a vital assurance 
prior to undertaking any more formal 
outcome evaluation. (A more complete 
discussion of evaluation is included in 
stage 5.) 
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Establish Process Evaluation 
Moasuras 

To help avoid overall program 
dysfunction because specific tasks 
aren't working, you should make sure 
that progrann checks are in place. 
Mechanisnns in place should track: 

■ work performed, time schedules and 
expenditures (internal measures) 

■ publicity, promouon and other 
outreach 

■ participation, inquiries or other 
responses 

■ functioning and quality of response 
systems (distribution, inquiries 
response) 

■ interim changes of audience 
awareness, knowledge or actions. 

Some ways of tracking include: 

■ weekly materials inventory review 

■ clipping services of print media 
coverage 

■ "bounceback" cards or followup 
phone calls with television and radio 
stations 

■ monitoring logs of television/radio 
stations for frequency and time of 
PSA airings 

■ monitoring volume of inquiries and 
length of time to reply 



■ reviewing telephone responses for 
accuracy and appropriateness 

■ checking distribution points to 
assess materials use (and make 
sure that materials are still available) 

■ phone calls or meetings with parti- 
cipating organizations to review 
progress and problems 

■ focus groups or telephone interviews 
with program participants/target 
audience members 

■ followup with teachers, physicians or 
other community professionals to 
check their preparedness, interest 
and to identify problenns. 

Work With intarmediaries 

You may have planned to use just one 
or a combination of media, interper- 
sonal and community channels for 
your program. However, once your 
program is visible, others may be 
willing to help. >fau may have 
opportunities to expand the number 
and kinds of channels involved. 
Although how your program expands 
depends upon your resources and 
communication strategies (stage 1), 
these opportunities can make program 
implementation exciting and chal- 
lenging. Some program managers may 
believe tha. he most creative (and 



rewarding) step is materials develop- 
ment. Actually, creative use of 
messages and materials is essential to 
success, creates tangible evidence of 
progress and can be very rewarding. 

Working with organizations or indi- 
viduals outside of your own agency is 
almost always necessary to reach a 
target audience These organizations 
(eg., a television station, hospital, PTA) 
or individuals (e.g., a family physician, 
pharmacist, visiting nurse) are *'inter- 
mediary** channels to reach the target 
audience. 

"Intermediaries*' can help you by 
providing: 

■ access to a target audience 

■ r credibility for your message or 
program because the target audi- 
ence considers them to be a trusted 
source 

■ additional resources — either tan- 
gible or intangible (eg., volunteers) 

■ added expertise (eg., training 
capabilities) 

■ co-sponsorship of community 
events. 

Although the Office of Cancer Commu- 
nications has found using intermedi- 
aries essential to fulfilling their 
program expectations, there are 
drawbacks that you should recognize 
and prepare foi. Working with other 
organizations can: 

■ be time consuming — to locate, 
convince them to work with you, 
gain internal approvals, undergo 
planning and/or training 

■ require altering your program— 
every organization has different 
priorities and perspectives, and 
intermediaries may want to make 
minor— or major — program 
changes to accommodate their 
structure or needs 

« result in loss of "ownership" and 
control — because other organiza- 
tions may change the time schedule, 
functions, or even the messages, 
and take credit for their part (or all) 
of the program. 
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You should be aware of these 
possibilities and be prepared to decide 
how much "give and take" you will be 
able to allow without violating the 
integrity of your program, its direction, 
and your own agency procedures. 
Balancing these decisions will require 
strategic thinking. You — or the staff 
person responsible for working with 
intermediaries — should also be: 

■ a good manager, to balance all 
progrann components 

■ a team player, to work with these 
organizations 

■ diplomatic and willing to negotiate 

■ willing to share credit for success. 

ChooM lnttrmediaii«t 

Working with other organizations is 
time consuming as well as profitable. 
In order to make the best use of your 
staff time available, follow these steps: 

■ nnake a list of potential internnediary 
organizations 

■ find out which agencies would reach 
the most of your audience 

■ consider which might have the 
greatest influence with your 
audience 

■ think about which organizations you 
are most likely to be able to con- 
vince to work with you (e.g., where 
you know a person to contact) 

■ think about which would require less 
support from you (e.g., fewer of your 
resources) 

■ prioritize list of organizations accord- 
ing to the factors listed above 

■ nnake contacts in priority order 
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Consider Working witli 
Business 

Cooperative ventures with the busi- 
ness sector (for-profit companies) can 
also help your program work, with 
benefits accruing to your program and 
the company. 




Private sector companies with an 
interest in health information include* 

■ supermarkets 

■ pharmacies 

■ insurance companies 

■ hospitals and HMOs 

■ producers of pharmaceuticals, foods 
and other health-related products. 



Other companies may be interested in 
working with you even if their product 
or service is not related to your 
program to: 

■ provide a useful public service 

■ improve their corporate image and 
credibility 

■ attract the attention of a particular 
sector of the public. 

In addition to providing funds, you 
might ask for these kinds of help: 

■ distribution of materials to their 
consumers (e.g., in a supermarket) 
or to their employees 

■ access to free or paid media space 
and time 

■ corporate endorsement and help in 
securing support from other 
corporations 

■ access to their market researc h or 
othe'' data 

■ use of their staff time or facilities 

■ sponsorship of related activities for 
their employees or for the public 

■ printing, mailing, production facilities 
or other in-kind contributions. 

Review and Revise Program 
Components 

Whether or not you continue to expand 
and involve more media outlets or 
organizations in your program, you 
should periodically assess whether. 

■ activities are "on track" and on time 

■ the target audience is being reached 

■ some strategies appear to be more 
successful than others 

■ some aspects of the program need 
more attention, alteration or 
elimination 

■ time schedules are being met 

■ resource expenditures are accept- 
able 





The process evaluation and other 
tracking measures you established 
should permit this assessment. You 
should establish specific intervals to 
review progress. Preparing progress 
reports— with successes, modified 
plans and schedules— can help you 
keep all your agency and program 
"players" informed and synchronized. 
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The process measures discussed 
in stage 4 are designed to nnon- 
Itor the progrann in progress. 
Tracking the nunnber of nnaterials dis- 
tributed, meetings attended or articles 
printed will tell you how the program is 
operating, and may tell you whether 
the target audience is responding; 
these measures will not tell you about 
the program effects: whether the 
target audience learned, acted or 
made a change as a result. Therefore, 
It is important to evaluate the results of 
your program— its effect or outcome. 

Most outcome measures are de- 
signed to tell you vjhat effect was 
achieved, but not how or why — these 
are the subjects of formative research 
and process measures. The effect or 
outcome is paramount, but you also 
need to know what happened, how 
and why which elements worked, and 
to analyze what should be changed in 
future programs. Therefore, plans for 
outcome measures are combined with 
other evaluation strategies during 
stage 1 planning. A review of all of the 
kinds of evaluation discussed in this 
guide can be found on page 64. 

Outcome Evaluation 

Outcome evaluation methodologies 
usually consist of a comparison 
between the target audience aware- 
ness, attitudes and/or behavior before 
and after the program. Unlike the 
pretesting methods ("formative 
evaluation") described in stage 3, 
these are quantitative measures, 
necessary to draw conclusions about 
the program effect. Going a step 
beyond process measures, outcome 
evaluation should provide more 
information about value than quantity 
of activity. The measures may be self- 
reported (e.g., interviews with the 
target audience) or observational (e.g.. 
changes in clinic visits or disease 
morbidity). Comparisons between a 
control group (one that did not receive 
the program, but is similar in other 
respects to the target audience) and 
the target audience receiving the 
program are desirable. 



ImiMictttiidlat 

As discussed in the introduction to this 
guide, communication programs are 
one contributor to the improvement of 
the public's health. In a "real world" 
environment, there are many factors 
which influence an individual's health 
behavior, including peer support and 
approval, self-esteem and other 
individual characteristics, advertising 
and mass media coverage of health, 
community arid institutional factors 
(such as the availability of services). It 
is usually extremely difficult to sep 
arate the impact of your communica- 
tion program from the effects of other 
factors ("confounding variables") on 
an individual's behavior. For this 
reason, impact studies are rarely 
initiated as a part of programs using 
only communication strategies, 

Datartnina What Evaluation to Do 

Limited resources may force you to 
choose between process evaluation or 
outcome evaluation. Neither, independ- 
ently, will provide you with a complete 
picture of what happened. Some 
experts will tell you that if you must 
choose, you should choose outcome 
evaluation— the only way to certify 
that you accomplished your objectives. 
However, process evaluation can help 
you understand w/)/ you did or did not 
accomplish your objectives. Therefore, 
others will advise that process meas- 
ures are more important — to allow you 
to manage your program well. 

Every program planner faces 
constraints to undertaking evaluation 
tasks, just as there are constraints to 




designing other aspects of a communi- 
cation program. These constraints may 
include: 

■ limited funds 

■ limited staff time and capabilities 

■ length of time allotted to the 
program 

■ limited access to computer facilities 

■ agency restrictions to hiring 
consultants or contractors 

■ policies limiting the ability to gather 
information from the public 

■ management perceptions regarding 
the va^ of evaluation 

■ levels of management support for 
well-designed evaluation activities 

■ difficulties in defining the objectives 
of the program, or in establishing 
agency consensus 

■ difficulties in designing appropriate 
measures for communication 
programs 

■ difficulties in separating the effects 
of program influences from other 
influences on the target audience in 
^'real world" situations. 

These constraints make it neces- 
sary to accommodate to existing limi- 
tations as well as the requirements of 
a specific program. However, it is not 
true that "something is better than 
nothing." If an evaluation design, data 
collection or analysis must be com- 
promised to fit limitations, the program 
must make a decision regarding 
whether: 

■ the required compromises will make 
the evaluation results invalid 

■ an evaluation strategy is essential 
for the particular situation, com- 
pared with other compelling uses for 
existing resources. 



These are some of the questions 
you should consider before deciding 
what kind of evaluation will be best for 
your program: 

■ How long will your program last? Will 
the implementation phase be long 
enough to permit measurement of 
significant effects and periodic 
adjustment? 

■ Do you want to repeat or continue 
your program? 

■ Are your objectives measurable in 
the foreseeable future? 

■ Which program components are 
most important to you? 

■ Is There nnanagement support or 
public dennand for program 
accountability? 

■ What aspects of the program fit best 
with your agency's priorities? 

■ Will an evaUation report help 
communication efforts compett? vVuh 
other agency priorities for future 
funding? 

There are a number of sources for 
you to find help when you design an 
evaluation. Several pertinent texts are 
included in the suggested readings at 
the end of this chapter. If there is not a 
planning and evaluation staff in your 
agency you may find help at a nearby 
university. Also, you may contact an 
appropriate clearinghouse or Federal 
agency and ask for evaluation reports 
that may have been prepared (but 
generally are not published) on similar 
programs. 



A frequent response to ''What kind 
of evaluaticxi measures are you plan- 
ning?** is "I don't have eriough money 
for evaluation." Rarely does anyone 
have access to resources for an ideal 
health comnnunication program, much 
less an ideal evaluation component. 
Nevertheless, there are practical 
benefits to including evaluation as a 
part of your work: to tell you whether 
your progrann is on track and how well 
it worked. The previous discussion of 
process, outcome and impact evalua- 
tion included some examples of the 
kinds of questions you might ask. With 
a little creative thinking, you will find 
that you can include some form of 
evaluation for almost any size of 
budget. The chart below gives ex- 
amples of evaluation tasks you might 
consider if you don't really have an 
evaluation budget ("minimal re- 
sources*'), and if you have a nrxxjerate 
budget for evaluation. It also gives you 
examples of the kinds of evaluations 
you might kJeally consider ("substan- 
tial resources"). The matrix is additive 
from left to right. That is, each ascend- 



ing program level could be expected to 
include the evaluation technique 
described at lower levels in addition to 
those described at the higher level. 

Efwnwits of an Evaluation 
Dtsign 

Every formal design, whether forma- 
tive, process, outcome, impact or a 
combination, must contain certain 
basic elenDorus. T** 3e include: 

t A Statement of Communication 
Obi^tives 

Unless there is an adequate definition 
of desired achievements, evaluation 
cannot measure them. Evaluators 
need clear and definite objectives in 
order to measure program effects. 

2. Definition of Data to be Collected 
This is the determination of what is to 
be me'isured in relation to the objec- 
tives. 

3. Methodology 

A study design is formulated to permit 
measurement in a valid and reliable 
manner 



4, Instrumentation 

Data collection instruments are de- 
signed and pretested. These instru- 
ments range from simple tally sheets 
for counting public inquiries to 
complex survey and interview forms. 

5, Data Collection 

The actual process of gathering data. 

6, Data Processing 

Putting the data into usable form for 
analysis. 

7, Data Analysis 

The application of statistical tech- 
niques to the data to discover signifi- 
cant relationships, 

8, Reporting 

Compiling and recording evaluation 
results. These results rarely pronounce 
a program a conplete success or 
failure To some extent all programs 
have good elements and bad. It is 
important to appreciate that lessons 
can be teamed from both if results are 
properly analyzed. These lessons 
should be applied to altering the 
existing program or as a guide to 
planning new efforts. 
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Apply What You Have Uamed 

The ideal way to apply evaluation 
findings is to impro • 'our ongoing 
program. M)u also r. use what yc ' 
learn from process or outcome evalua- 
tion measures to: 

■ justify your program with manage- 
ment 

■ provide evidence of need for addi- 
tional funds or other resources 

■ increase institutional understanding 
of and support for health cc<nmuni- 
cation activities 

■ encourage ongoing cooperative 
ventures with other organizations. 

Revise Program 

If your program is continuing or you 
have an opportunity to advise others 
who may plan similar programs, take 
the time to apply what you have 
learned. For example. 

■ Reassess goals and objectives 

■ Has anything changed (e.g.. with 
the target audience, tho commu- 
nity, or your agency's mission) to 
require revisions in the original 
goals and objectives? 

■ Is there new information about the 
health issue that should be in- 
corporated into the program mes- 
ages or design? 

■ Determine areas where additional 
effort is needed 

■ Are there objectives ihat are not 
being met? VJh/> 

■ Are there strategies or activities 
that did not succeed? Are more 
resources required'^ Do you need 
to review why they didn't work and 
what can be done to correct any 
problems? 

■ Identify effective activities or strate- 
gies 

■ Have some objectives been met 
as a result of successful activi- 
ties? 

B Should these be continued and 
strengthened because they appea*' 
to work well? 



■ Or should they be considered suc- 
cessful and connpleted? 

■ Can they be expanded to apply to 
other audiences or situations? 

• Compare costs and results of dif- 
erent activities 

w What were the relative costs 
(including staff time) and results of 
different aspects of your program? 

« Are there some activities that 
appear to work as well but cost 
less than others? 

« Reaffirn) support for the program 
« Have you shared the results of 

your activities with the leadership 

of your agency? 
« Did you remember to share this 

information with the individuals 

and organizations outside your 

agency who contributed? 
« Do you have evidence of program 

effectiveness and continued need 

to convince your agency to 

continue your progranY? 

■ Do you have new or continuing 
activities that suggest the involve- 
ment of additional organizations? 

■ Determine to end a program that did 
not work. 

Share What You Have Learned 

It is frequently difficult to find the time 
to analyze and report on what you 
have learned and share it with others. 
You may find that other responsibilities 
leave you little time to prepare fornnal 
documentation of your program or to 
submit findings for publication. Never- 
theless, what you learn from imple- 
menting a communication program 
might be invaluable to someone who is 
faced with a similar responsibility. You 
may not have to prepare a formal 
report or article to let others know 
what you have learned. Consider. 

■ letters about your findings to appro- 
priate medical, public health or 
health education journals 

9 a ooster presentation at a relevant 

professic^al meeting 
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■ a program description and sample 
materials sent to a related clearing- 
house, Federal or state agency 

■ local professional newsletters 

■ letters, phone calls, brief reports or 
nneetings with your peers in similar 
organizations. 

Letting others know about your 
program may prompt them to tell you 
about similar experiences, lessons, 
new ideas or potential resources. 
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National Sources of 
HeaKh-RelatedData 

The National Center for Health 
Statistics (NCHS) 
U.S. Public Health Service, NCHS 
3700 East-West Highway 
Hyattsville, MD 20782 

NCHS tracks and analyzes changes in 
health status in the U.S. A sumnnary of 
status and trends is published each 
year as Health/United States. 

Morbidity and Mortality Weekly Report 
(MMWR) 

Centers for Disease Control 
Atlanta, GA 30333 

The MMWR provides immediate alerts 
regarding disease incidence, morbidity 
and mortality as reported to CDC by 
health departments nationwide. 

The Roper Center 
P.O. Box ^40 
Storrs, Cr 06268 

The Roper Center collects and stores 
public opinion data collected by survey 
organizations including NORC, Gallup, 
Roper, Harris, Opinion Research 
Corporation, Yankelovich, Gordon 
Black Company, various news polls 
and special studies. 

The ODPHP National Health 
Information Center 
RO. Box 1133 

Washington. DC 20013-1133 
(800) 336-4797 (from outside of the 
Washington Metropolitan area) 

The center helps the public locate 
health information through the 
identification of health informaiion 
resources and a referral service. 



Journals 

These journals cover health communi- 
cations p»'ogram development and 
related topics discussed in this book: 

Alcohol Health and Research World, a 
quarterly publication of the (DHHS) 
National Institute on Alcohol Abuse 
and Alcoholism, ADAMHA, 5600 
Fishers Lane, Rockville, MD 20857. 

American Demographics, Box 68, 127 
West State St.. Ithaca, NY 14851. 

American Journal of Health Promotion, 
a quarterly publication. Box 1287, 
Royal Oak, Ml 48068. 

American Journal of Public Health, a 
monthly publication of the American 
Public Health Association, 1015 15th 
St. NW, Washington, DC 20005. 

Evaluation and the Health Professions, 
quarterly, Sage Publications, Inc., 2111 
W. Hillcrest Dr., Newbury Park, CA 
91320. 

Health Education, bimonthly a 
publication of the American Alliance 
for Health, Physical Education, 
Recreation and Dance, 1900 
Association Drive, Reston, VA 22091. 

Health Education Quarterly a 
publication of the Society for Public 
Health Education, John Wiley & Sons, 
Inc., Periodicals Division, 605 Third 
Avenue, New York, NY 10158. 

Health Education Research, quarterly, 
IRL Press Inc., PO Box Q, McLean, 
VA 22101-0850. 

HealthLink, a quarterly publication of 
the National Center for Health 
Education, 30 East 29th St., t ^w York, 
NY 10016. 

Journal of Communication, quarterly 
publication by Oxford University Press, 
University of Pennsylvania, 3620 
Walnut St., Philadelphia, PA 
19104-3858. 



Public Health Reports, a bimonthly 
publication of the U.S. Public Health 
Service, Dept. of Health and Human 
Services, Hubert Humphrey BIdg., Rm. 
721-H, 200 Independence Ave. SW, 
Washington, DC 20201. 

Public Opinion Quarterly a quarterly 
publication of the American 
Association for Public Opinion 
Research, University of Chicago Press, 
5801 S. Ellis Ave., Chicago, IL 60637. 
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The SMOG Readability Fbrmula 

To calculate the SMOG reading grade level, 
begin with the entire written work that is being 
assessed, and follow these four steps: 

1. Count off 10 consecutive sentences near the 
beginning, in the middle, and near the end of 
the text. 

2. From this sample of 30 sentences, circle all 
of the words containing three or more 
syllables (polysyllabic), including repetitions of 
the same word, and total the number of words 
circled. 

3. Estimate the square root of the total number 
of polysyllabic words counted. This is done by 
finding the nearest perfect square, and taking 
its square root. 

4. Finally, add a constant of three to the square 
root. This number gives the SMOG grade, or 
the reading grade level that a person must 
have reached if he or she is to fully under- 
stand the text being assessed. 

A few additional guidelines will help to clarify 
these directions: 

• A sentence is defined as a string of words 
punctuated with a period (.), an exclamation 
point (!) or a question mark (?). 

• Hyphenated words are considered as one 
word. 

• Numbers which are written out should also be 
considered, and if in numeric form in the text, 
they should be pronounced to determine if 
they are polysyllabic 



• Proper nouns, if polysyllabic, should be 
counted, too. 

• Abbreviations should be read as 
unabbreviated to determine if they are 
polysyllabic 

Not all pamphlets, fact sheets, or other 
printed materials contain 30 sentences. To test a 
text that has fewer than 30 sentences: 

1. Count all of the polysyllabic words in the text. 

2. Count the number of sentences. 

3. Find the average number of polysyllabic 
words per sentence as follows: 

nv/pranp - Total # of polvsvllabic words 
average - lotai f oT sentences 

4. Multiply that average by the number of 
sentences short of 30. 

5. Add that figure on to the total number of 
polysyllabic words. 

6. Find the square root and add the constant of 
3. 

Perhaps the quickest way to administer the 
SMOG grading test is by using the SMOG 
conversion table. Simply count the number of 
polysyllabic words in your chain of 30 sentences 
and look up the approximate grade level on the 
chart. 

An example of how to use the SMOG 
Readability Formula and the SMOG Conversion 
Table is provided on the following page. 
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Example Using the 
SMOG Readability Formula: 



Sample only: Information may not be current. 





vThe key is actionXVbu can help protect yourself against 
cancer) Act promptly to: 

f ^Prevent swne cancers through simple changes in 
lifestyleJ 

(^'Fmd out about early ^etectiop ) tests in your home,) 
(^'Gain peace of mind through ^^^K^di^pbheckups.) 

Cancers Yoo ShouM Know About 

(7. Lung Cancer is the number one canc^^jc^qng men, both 
injhe number of new cases each ycai(^io^and deaths 
" grates are due mainly to 
[ smoking, you can largely 
premit lung cancer^^The ris k is red uced by smoking less, 
and by using lower tar and ipcotiia brands) But quitting 
altogether is by for the most cffecth« safeguard. The 
American Cancer Society offers Quit Smoking Clinics 
and self-help materials. 

Colorectal Cancer is second in cancer deaths (25,100) 
and third in new cases (49,000). When it is found early, 
chances of cure are good. A regular general pt^cal 
usually indudes a digital examination of the rectum and a 
guaiac slide test of a stool spedmen to check for invisible 
blood. Now there are also Do-It-Yourself Guaiac Slides 
for home use. Ask your doctor about them. After you 
reach the age of 40, your r^^ular check-up may include a 
"Procto,** in which the rectimi and part of the colon are 
inspected through a hollow, lighted tube. 

0^ Prostate Cance r is second in the num ber of n ew cases 
each year ^,00 p), and third in deaths QOy^& f^C^bcairs 
mainly in men over 60j(^' <^u]§ > r ectal ca m of tl 
prostate by your doctor is the best j ^tect^ ) 
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AChcck-UpPliysOff 

i&Be sure to have a g5iei3 J( ghysi^ including an 
oral exani/u is your best goaran^ of good health) 

*This pamphlet is from the American Cancer Society. 
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floiw Cancer Works 
**If we kno wsomethi ng about how cancer works, we can 
act more tffec ^tiv^ to ^yotect oursdves against the 
disease) Here are the basics. 

(*^1. Cancer spr gds; tim e coum&TCancer is i gicontrolliS 
growth of ^norn^ .Qdl sXin)Cg ins small and if 
unchecked* spread^Uf *4(^cte^ in an early, local stage, 
the chances for cure are best«^ 

Risk j Screasg with age— This is not a reascH i to wor ry, 
but a signal to have morefeguIaH thorough( ghysl<^ 
check-ups) Your doctor or clinic can advise you on 
what tests to get and how often they should be 
performed. 

What you can do— Don't smoke and you will sharply 
reduce your chances of getting lung cancer. Avoid too 
much sun, a major cause of skin cancer. Learn 
cancer's Seven Mi^ming Signals, listed on the back of 
this leaflet, and see your doctor promptly if they 
persist. Pain usually is a late symptom of cancer; don't 
wait for it. 

Unproven Remedies 

Beware of unproven cancer remedies. Th^ may sound 
appealing, but they are usually worthless. Relying m 
them can dday good trea toient until it is t oo late( (3ieck 
with your doctor or the ^hen^ Cancer ggdety ) 

More Infori Mtfott 

(}H6t more i ftfSnn ati^ of any kiiKl a bout cai^ — free of 
^t— contact yoiSlocal unit of th^ ^eric^ Cancer 
ie9^ 



Know Cancer's Seven Wmung Signals 

11^ 1. Change in bowd or bladder h^its.) 
(24. 2. A sore that does not heal.) 
(-35. 3. Unusual bleedmg or dischaige.) 
^2U' 4. TThickcning or tomp in b rea st or else where) 
^27.5. lodigest^ or d Lfficulfl ^ in ^^dwiB g.) 
^20. 6. o&w5g >change in wart or mole.) 
Nagging cough or hoarseness.) 

(3d. If yoo have a warning s^pal> see your docton) 
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Wb have calculated the reading grade level 
for this example Compare your results to ours, 
then check both witli the SMOG conversion 
table: 



Readability Test Calculations 




Total Number of Polysyllabic Wonls 


= 38 


Nearest Perfect Square 


= 36 


Square Root 


= 6 


Constant 


= 3 


SMOG Reading Grade Level 


= 9 



SMOQConMraionTabIt* 



Total Polysyllabic 
Word Counts 



Approximate Grade 
Level (±1.5 Grades) 



0-2 

3-6 

7-12 

13-20 

21-30 

31-42 

43-56 

57-72 

73-90 

91-110 

111-132 

133-156 

157-182 

183-210 

211-240 



4 
5 

e 

7 
8 
9 
10 
11 
12 
13 
14 
15 
16 
17 
18 



•Developed by Harold C McGraw, Office of Educational Research, 
Baltimore County Schools. Towson, Maryland 




■ Educational Materials Review Forms 

■ Pre-Post Booklet Testing Form (Self-Administered) 

■ Central Location Intercept Questionnaire 

■ Focus Group Moderators Guide 

■ Gatekeeper/Professional Review Questionnaire 

■ PSA Pretest Questions 
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Educational Materials Review Forms 

Source: Office for Substarjce Abuse Prevention, ADAM HA 



I. PRODUCT DESCRIPTION 

Title of Product 

Producer/Author 



Organization 
Address 



_ Publication Date 
Contact 



Accession No 



Date 



Screener 



Phone . 



other Sponsors/Endorsers 



Format: 




_ Fact Sheet 


__PSA 


Brochure 


Magazine 


_ Booklet 


Article 


__Book 


Pertornnance 


Poster 


Taped Message 


_ Print Ad 


Newsletter 


_ 3/4" Video 


_ Contest 


_VHS Video 


Workbook 


I6mm Filnn 


Curriculum 


_ Slides 


Classroom Material 


__ Audiotape 


Comic Book 


Script 




_ Information Package 




_ Software 




_ Other (describe) 





Length: 

Context(S): (check all that apply) 
_ Part of a Program (describe) 



_ With Other Materials 
_ Stands Alone 

Has Training CorDponent (please enclose) 

Topic(S): (check all that apply) 
_ Alcohol 
^ Drugs 

_ Drug (specify) 

Both Alcohol and Drugs 
_ Awareness 
_ Prevention 

Intervention 

Treatment 

_ General Health/Safety 
_ Other (describe) 



Mode(s) of Delivery: (check all that apply) 
^ Individual 
Group 

Instructor-led 
Mass Media 
_ Self-Instructional 

Setting<S): (check all that apply) 

Home 

School 
___ Community 

_ Health/Mental Health Care 
_ Worksite 

Justice System 

Social Services System 

Mass Media 

Other (describe) 



Target Audience<s): (check aii that apply) 

_ General Public 

_ Parents (specify age of child) 

„ Blacks 
_ Hispanics 

Native Americans 

_ Asian and Pacific Islanders 

_ Health Care Providers (specify specialty) 



_ High Risk FamiliesAbuth 

Policymakers/Administrators 

_ Youth (specify ages) 
_ Young Adults (18-25 years) 
_ Educators (s'secify grade{sD 
_ A/D Treatment Professionals 
_ A/D Prevention Professionals 
Employers 

_ Scientists and Reseachers 
Other (describe) 



Language{s): (check all that apply) 

English 

Spanish 

_ Bilingual (specify) 

Other (indicate) 

Readability: 

Ijow Literacy (grade level 3-5) 

_ Very Easy (grade level 67) 

^ Easy (8) 

_ Average (9-10) 

_ Fairly Difficult (1M3) 

Difficult (14-16) 
_N/A 

Pretested/Evaluated: 

Yes Unknown 

_ No 

(If yes, describe and include copy of report if 
possible)^ 



Current Scope: 

National 

Regional 

_ State 

_ Local 

Availability: 

Unknown 

Restrictions on Use 

_ Permission Required to Reproduce 

_ Citation Required 

Available Free 

_ Negatives Available on Loan 
_ Payment Required 

_ Prices 

_ Available Through Free A-V Loan Program 
Source (if different from above) 



Description: (Please describe the product in two-three sentences ) 



Comments: 
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IL FIELD REVIEW FORM 



ACC No ^ 

Date 

Reviewer „ 



Target Audience (if different from Screening Form) 
Topic (if different from Screening Form) 
Major Messages (list) 



Persuasive Technique (describe) 



Distinguishing Qualities (describe): 





Excellent 






Poor 


Production Quality 


5 A 


3 


2 


1 


(Comments) 










Content 


5 4 


3 


2 


1 


(Comments) 










Credibility 


5 4 


3 


2 


1 


(Comments) 










Ability to Attract Attention 


5 4 


3 


2 


1 


(Comments) 










Ability to Convey Information* 


5 4 


3 


2 


1 


(Comments) 










Ability to Change Attitudes 


5 4 


3 


2 


1 


(Comments) 










Ability to Elicit Appropriate Action 


5 4 


3 


2 


1 



(Comments) 



Appropriate for National Distri^- • /ion Yes _ No Limited Use (describe) 

(Comments) 



Overall Rating 5 4 3 2 1 

(specify any particular strengths/weaknesses) 

Recommend for further consideration (eg , promotion, replication, purchase, adaptation, testing or evaluation)'^ 
Yes No 

Please explain reconimendation 
Return to* 
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Considerations for Reld Review 

1. Target audience— What audience is the material best suited for? For whom should it not be used? Consider the 
language style, use of terminology, length, appropriateness of examples and format in determining the target 
audienca 

2. Re^'suasive technique— Are the messages positive and 'jpbeat? Are positive role models used? Fear appeals? 
Authority figures (who)? Peer pressure? 

3. Distinguishing qualities — Innovative or unique presentation, format or style? Fills a need for specific audience or 
message? 

4. Production qualities — Is the material professional in appearance, attractive, well-written? Is the production format 
appropriate for the intended use (e.g.. setting, equipment required)? Should production changes be considered 
(ag.. use of less or more color)? 

5. Content— Clear and accurate? Up to date? Appropriate message, tone and appeal? Stimulating? New knowledge? 
Perpetuate myths or stereotypes? Balanced and credible? Biased or judgmental? 

6. Elicit action— Describes desired behavior? Illustrates skills required? Demonstrates appropriate behavior? 

7. Credibility— Is production or distribution source credible for target audience? For intermediahes (ag.. teachers or 
parents)? Is message, theme, presentation credible? 

8. Appropriate for national distribution— Will materials Stan':' alona or require aining for use? Inappropriate for some 
audiences (ag.. culturally inappropriate) or geographic areas? 

9. Recommendation for evaluation— Are there questions or uncertainties that need to be resolved pnor to determin- 
ing disposition? Should materials be tested? 
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Pre-Post Booklet Testing Form 

Source" U S Environmental Protection Agency 



1. Pretest Questions 

As you probably are aware. Toms River is the site of a pilot project designed to inform residents about 
potential environmental hazards associated with the Superfund site, and to encourage their involve 
ment in EPA's decision-making process for cleanup of the site. 

We would appreciate your willingness to share your reactions to the attached fact sheet by reading 
It and answering a few questions. We do not ask your name and all inform ition you provide will re- 
main confidential. 

Because only a few Toms River citizens are being asked to help judge \h\s matprial, your response 
is particularly valuable. 

Before you begin, please check the appropriate answers to these four questions 

1. How much would you say you know about the Toms River Superfund study'' 
A little Some A lot 

2. Is there anyth'ng in particular you want to know about the study'' 
Yes No 

If yes, please specify. 

(Note: more knowledge questions can be added here.) 
3 Are you or any member of your family an employee/former employee of (Superfund site company)'' 
Yes No 

4. Are you a member of any group particularly concerned about the environment 

Yes No 

Now, please turn the page and read the fact sheet. 
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il. Posttest Questions 

Now that you have finished reading the fact sheet, please answer the questions beiov^ you mdy r^tf r 
back to the fact sheet as you consider your response if you wish 

1 In your own words, what would you say is the purpose of the Superfund study'^ 
(Note: additional l^nowledge questions can be added here) 

2. How nnuch of the information in the fact sheet was new to you'^ 
Most ot it Sonne of it None 

3 Do you have questions about the Superfund study which were not answered m the fad shee!^ 
Yes No 

If yes, please list. 

4 Was there anything you particularly hked about the fact sheef^ 
Yes No 

If yes, whai'^ 

5 Was there anything you particularly disliked about the fact sheet, or found confusing ' 
Yes No 

If yes, whai'^ 

6 This fact sheet is most appropudie for (ch?ck all that apply) 
General public College graduates Professionals 

7 Would you recommend the fact sheet to a friend or family member'^ 
Yes No 

8 The following are a senes of pnrases describing the fact sheet Please circle the one L^dce 
each line that most closely reflects yoar opinion 



a 


very interesting 


somewhat interesting 


not at all interestma 


b 


very informative 


somewhat informative 


not informative 


c 


accurate 


partially accurate 


inaccurate 


d 


very clear 


somewhat clear 


confusing 


e. 


very useful 


somewhat useful 


not useful 


f 


unbiased 


biased towards government 


biased towards mdust^' 


g 


easy to read 


understandable 


hard to understand 


h 


complete 


scmewhat complete 


incomplete 



9 Would you like to say anything else about the fact sheet"^ Please comment 



Thank you very much for your help in reviewing this fact sheet 
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Central Location Intercept (Respondent s name and ID No) 

Questionnaire (Location Houston = 1. Chicago = 2) 

Source National Heart. Lung and Blood Institute 

(Phone Number) 



introduction 

Good afternoon (evening). 

My name is (interviewer's initials) , and I'm with Quick Test, a marketing research firm. 

We are conducting a research study in this mall with people who are between 25 and 55 years of age 
Would you be willing to spend a few minutes answering some questions'' 



Screener 

S 1 What IS your age? (RECORD) 

1 Under 25 (THANK AND TERMINATE) 
2. Between 25 and 40 (CONTINUE) 

3 Between 41 and 55 (CONTINUE) 

4 Over 55 (THANK AND TERMINATE) 



S 2 Have you ever been told you have any of the following^ 

YES NO 

1 Diabetes CONTINUE CONTINUE 

? High blood pressure CONTINUE TERMINATE 

3 High cholesterol CONTINUE CONTINUE 

S 3 Which of the following best describes the highest level Oi' education you've completed (CIRCLE 
ONE) 

1 Elementary or grade school (CONTINUE) 

2 Some high school (CONTINUE) 

3 High school graduate (CONTINUE) 

4 Some college (CONTINUE) 

- College graduate (CONTINUE— MAXIMUM OF 2 PER SUBGROUP) 

6 Some g-aduate work (THANK AND TERMINATE) 

7 Graduate degree ( THANK AND TERMINATE) 



84 Respondent's sex 

1 Male 

2 Female 



S 5 Respondent's race 

1 White (CONTINUE) 

2 Black (CONTINUE) 

3 Other (THANK AND TERMINATE) 
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S.6 I would like to show you three television messages which have been produced as a public 

service and get your reactions to them. This will not take more than 15 minutes and we would 
like to of^er you an appreciation of $2.00 as a means of saying "thank you" for your time. If you 
would like to see these public service messages, would you piease come with me into our 
viewing room? 

1. No (THANK AND TERMINATE) 

2. Yes (CONTINUE) 

NOTE: Participants are to be recruited in the following subgroups. 

15 black males, ages 25-40 
15 white male.s, ages 25-40 
15 black males, ages 41-55 
15 white males, ages 41-55 
15 black females, ages 4 1-55 
15 white females, ages 41-55 

In each of the subgroups listed above, one-third of the respondents should follow Order 1. one-third 
should follow Order 2, and one-third should follow Order 3, as specified below: 

ORDER 1. Show "Randy" first and administer questionnaire. Then show "Red Ballocp" and 

administer questionnaire, followed by "Wild Horse" and questionnaire Thank and pay $2 
appreciation. 

ORDER 2. Show "Red Balloon" and administer questionnaire Then show "Randy * and administer 
questionnaire. Show "Wild Horse,** administer questionnaire, and thank and pay $2 
appreciation. 

ORDER 3. Show "Wild Horse" and administer questionnaire. Then show "Red Balloon" and adminis- 
ter questionnaire, followed by "Randy" and questionnaire Thank and pay $2 appreciation 

CHECK ORDER BEING USED: 

ORDER 1 ("Randy," "Red Balloon," "Wild Horse") 

ORDER 2 ("Red Balloon," "Randy," "Wild Horse") 

ORDER 3 ("Wild Horse," "Red Balloon" "Randy") 

FOR EACH RESPONDENT FASTEN TOGETHER THE SCREENER, "RANDY" QUESTIONNAIRE, 
"RED BALLOON" QUESTIONNAIRE AND "WILD HORSE" QUESTIONNAIRE IN THAT ORDER IN 
PREPARATION FOR COMPUTER ENTRY REMEMBER TO RECORD TIME OF FINISH AT BOTTOM OF 
SCREENER. 

Time start: 

Time finish: 
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Questionnaire ^"^r "Randy" 

IF ORDER 1. READ THE FOLLOWING. 

I'd like to show you tne first public service message and then ask you a few questions about it The 
message is still in rough form, so you will need to use your imagination a little to picture what it will 
look like after final production. In this version you will see drawings used to get the ided across, but in 
the final version there will be live action and it will have a more finished look Please try to concen- 
trate more on the concept than on the drawings as you watch them 

!F ORDER 2, READ THE FOLLGWiNQ- 

Now rd like to show you the second message and get your reactions Again, please remember that 
this is in rough form and is not yet in finished form 

IF ORDER 3, READ THE FOLLOWING 

I'd like to show you the third message now. Again. Tli be asking for your reactions Please remember 
that It is still in rough form. 

PLAY "RANDY" 

MOTE: Throughout the questionnaire, ^^HBP" should be read as "high blood pressure" 

1. What did you think was the mam idea of this announcement^ (RECORD FIRST MENTION) Anything 
else? (REC9RD ALL OTHER MENTIONS.) 



First Mention 
(Circle only one) 



Anything Else 
(Circle all other 
mentions that apply) 



8 
9 
10 
11 
12 



2 
3 
4 
5 
6 



high b!ood/HBP 

It's up to you— your responsiDility to take care of HBP 

If you don't take care of HBP. won't be around for other things 

Take care of yourself (general) 

Take care of HBP control HBP 

You can enjoy life (have a normal/active/iong life) by taking care of 
HBP 

Stay on HBP treatment, keeo up fight, work at ft don t ease up 

Take care of HBP for your loved ones 

Treat your HBP, do what you'' doctor says 

Control HBP or could become ill (stroke/die) 

Take care of your HBP every day. treat it for life 

Other 

RECORD - 



XX 



6 
7 
8 
9 

10 



2 
3 
4 
5 
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2. What, if anything, about the announcement did you particularly ///(e? (RECORD FIRST MENTION) 
Anything else'? (RECORD ALL OTHER MENTIONS) 



1 Everything 
2- Nothing 

3 Liked nnessage in general 

4. Easy to follow/understand 

5. Attention-getting/interesting 
6 Message is innportant 

7, Message contains useful infornnation 
8 Good rennmder 

9. Liked idea of having control/being able to do sonnething about HBP to 
help yourself 

10 Message is direct and to the DOint 

11 Focuses on blacks 

12 Message is drannatic 

13 Message speaks directly to nne 

14 Liked encouraging tone/can have a full life if treat HBP 

15 Message and pictures fit well together 

16 Don't know 

17 Other 

RECORD 



First Mention 
(Circle only one) 

1 
2 
3 
4 
5 
6 
7 
8 

8 

9 
10 
11 
12 
11 
12 
13 
XX 



Anything Else 
(Circle all other 



mentions 



hat apply) 



3- What, if anything, about the announcement did you particularly of/s///(e^ (RECORD FIRST MENTION) 
Anything else'? (RECORD ALL OTHER MENTIONS) 

Anything Else 



First Mention 
(Circle only -^le) 



(Circle 
mentions 



other 

:hat apply) 



1 


Everything 


1 


2- 


Nothing 


2 


3- 


Disliked message m general 


3 


4 


Message was difficult to follow/understand 


4 


5 


Voices hard to understand 


5 


6 


Too light/not serious enough 


6 


7 


Too serious/scary/overly dramatic 


7 


8 


Disliked black voice/image 


8 


9 


Not especially attention-getting or interesting 


9 


10 


Not enough information, information not new or useful 


10 


11 


Too per»simistic, grim 


11 


12. 


Message and pictures didn't fit well together 


12 


13 


Message didn't relate to me 


13 


14 


Don't know 


14 


15 


Other 


XX 
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4. Was there anything in the announcement that you found confusing or hard to understand'^ 
(RECORD FIRST MENTION) Anything else? (RECORD ALL OTHER MENTIONS) 



1 Nothing (everything was clear) 

2 Confusing in general 

3. Message not clear 

4. Words were hard to understand 

5. Didn't understand that the father had died 

6 Didn't understand why the father didn't live past fifty 

7 Too much information presented 

8 Not enough infornnation presented 
9. Message didn't relate to me 

10. Didn't underotand who was speaking 
11 Other 

RECORD 



First Mention 
(Circle only one) 

1 

2 

3 

4 

5 

6 

7 

8 

9 
10 
XX 



Anything Else 
(Circle all other 



mentions 



hat apply) 



5. Was there anything in the announcennent you found hard to believe'^ (RECORD FIRST MENTION) 
Anything else? (RECORD ALL OTHER MENTIONS) 



1 No. nothing 

2 Mesnage not believable in general 

3 Not believable that someone would die at fifty because didn \ take 
care of HBP 

4 Not believable that HBP is that dangerous to health 

5 Not believable that HBP 'S that hard to control 

6 other 

RECORD 



First Mention 
(Circie only one) 

1 

2 

3 
4 
c; 

XX 



Anything Else 
(Circle all other 
mentions that apply) 



6. (ASK ONLY FOR THOSE FOLLOWING ORDER 1) Can you tell me who the sponsor of this message 
was? 



1 National Cholesterol Education Progrann 

2 The government 

3. National Heart Association 

4 Don't remember 

5 Other 

RECORD . 



First Mention 
(Circle only one) 

1 

2 
3 
4 
XX 



Anything Else 
(Circle all other 



mentions 



hat apply) 
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7. Tm going to read to you a set of statements describing the message you just saw For each 
statement p'ease tell me whether you strongly agree, agree, neither agree nor disagree, disagree, 
or strongly disagree with the statement. (READ STATEMENTS AND SHOW SCALE) 



1 The message was interesting 

2 The message was convincing 
3- The message was irritating 

4 The message was confusing 

5 The message nnade its point 

6. The message was not serious enough 

7. The message was offensive 
8 The message was scary 

9. The message was believable 

10 The message gave nne useful infornnation 

11 The message gave useful infornnation for other 
people 

The message captured nny attention 
The message will capture the attention of those 
With HBP 

The message was a good renninder to take care 
of HBP 

15. The message had an overall encouraging tone 
^ The message was too mild, it should be stronger 
I will be nnore conscientious about nny HBP 
treatment 

Staying on nny HBP treatnnent progrann is a 
struggle for nne 

The message convinced nne that it s innportant 
to control HBP 



Strongly 
Agree 



12 
13 

14 



16 
17 

18 

19 



Agree 

2 
2 
2 
2 
2 
2 
2 
2 
2 
2 

2 
2 



2 
2 
2 

2 

2 

2 



Neither 
Agree Nor 
Disagree 

3 
3 
3 
3 
3 
3 
3 
3 



3 
3 



3 
3 

3 

3 



Disagree 

4 
4 
4 
4 
4 
4 
4 
4 
4 



4 
4 



Strongly 
Disagree 

5 
5 
5 
5 
5 
5 
5 
5 
5 
5 

5 
5 
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Focus Group Moderator's Guide 

Source. National Institute on Aging 



Warm-up and Explanation 

1. Introduction 

a Thanks for coming 

b Your presence is important. 

c Describe what a focus group is — 

like an opinion SL^ey, but very 

general, broad questions 

2. Purpose 

a We will be discussing some issues 

related to health 
b I'm interested in all of your ideas. 

comments, and suggestions 
c There are no right or wrong 

answers. 

d All comments— both positive and 

negative — are welcome 
e Please feel free to disagree with 

one another. We would like to have 

many points of view 

3. Procedure 

a Explain use of tape recorder (one- 
way mirror). All comments are 
confidential— used for research 
purposes only 

b I want this to be a group discussion, 
so you needn't wait for me to call on 
you Please speak one at a time, so 
that the tape recorder can pick up 
everything 

c We have a lot of ground to cover, so 
I may change the subject or move 
ahead Please stop me if you want 
to add something 

4. Self Introductions 

a Ask each participant to introduce 
himself/he^'self — tell the group your 
name, and where you live 



General Attitudes Health and Aging 

1. Is your health something that you 
actively think abouf^ (If not, why 
no\^) 

2. When you think of health, what do 
you think abouf^ 

3. What about the health of your 
family^ What issues or topics do 
you think about? 

4. (45-54 and over 55 groups) Do you 
think about growing older, and the 
changes that might bring'^ What do 
you think abouf^ (Probe re health 
changes independence, changes in 
lifestyle ; 

5. What do you think other people/your 
parents feel about growing older'? 
(Probe re health changes, inde- 
pendence, changes in lifestyle) 

Relationship to Older Family Members 

1. Do you have parents, other older 
relatives or friends who are close to 
you'? 

2. What IS your relationship with these 
older relatives/friends'? (How n^.uch 
contact, do you feel close to them'? 
Responsible'?) 

3. Do you think about their health and 
well-being'? What, specifically, do 
you think abouf? (Probe re* health 
concerns, independence, living 
situations ) 

Knov/ledge of Alzheimer's Disease 

(If Alzheimer's Disease nasn't come 

up in the conversation) 

1. Have you ever heard of Alzheimer's 
Disease'? What have you heard'? 

^. What do you think it is'? Is it different 
from just getting old'? (Probe for age 
of onset, how many people it af- 
fects, cause(s). symptoms, preven- 
tion, how It IS diagnosed, how it is 
treated, how quickly it progresses, 
IS It contagious, what happens, new 
research they have heard about, 
what kind of care is required^at 
home, nursing homes, related 
costs ) 



3. Is It (senility or Alzheimer's) an 
inevitable consequence of aging? 

4. Where have you heard about 
Alzheimer's Disease (probe for 
source)'? 

5. Do you have questions'? What 
would you like to know about it? 
What else? 

6. Do you ever have any thoughts or 
concerns about Alzheimer's 
Disease? For yourself or for 
someone else'? (Who'?) 

7. What kind of concerns'? 

Sources of Information (General) 

1. If you wanted to find out more 
about Alzheimer's Disease (had 
questions or concerns), how would 
you go about it? (Probe for. 
additional sources, e.g., family 
physician, friends, workplace 
source, where else in own 
community, library or national 
agencies.) 

2. Do you know of any agencies or 
organizations that deal especially 
with older people or Alzheimer's 
Disease'? 

Facing Alzheimer's Disease 

1. Now I want you to think about a 
particular sttiiation where you are 
concerned about a friend or 
relative You think they might have 
Alzheimer's Disease What would 
you do'? (Probe how would you 
interact with that individual [eg , a 
parent], where would you turn for 
help [call, write. goP) 

2. In particular, what would you need 
to know to deal with the situation'? 
Issues to probe for 

^1 more information about what it is 
r> diagnosis 

n consequences— what happens 
n treatment 

physician referral/or how to find 

one 

research they've heard about 
'1 services— where to go, what 
kinds of services exist 
financial burden 
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Sources of Information and Assistance 
(When Facing Alzheimer's Disease) 

1. Would you be more likely to seek 
help in your own community or to 
contact a national organization'^ 

2. What if your relative lived in another 
town'? What would you do then'? 

3. Have you ever heard of the National 
Institute on Aging'? 

4. How have you heard of them/What 
have you heard'? (If not it is a part 
of the National Institutes of Health 
in BethPsda, Maryland The NIH is 
the Federal agency respo^sib'e 
medical research ) 

5. Would you be likely to call or write 
to the National Institute on Aging'? 
(Why or why nof?) 

6. Have you ever heard of the 
Alzheimer s Disease and Related 
Disorders Association (ADRDA)'? (If 
not mentioned previously ) (If not. 
explain ADRDA and its chapters ^ 

7. How have you heard of it/What have 
you heard'? 

8. If you had questions would you be 
more likely to contact a national 
organization or a local organization'? 

9. Would you prefer to be able to talk 
with someone (eg . a "hotline ") or 
would you want written information'? 
Or would you prefer to speak with 
someone m person'? Wh/? 



Attitudes Toward Proposed Center 

1. The National Institute on Aging is 
planning to establish a National 
Alzheimer's Disease Center The 
Center will be a national source of 
information about Alzheimer's 
Disease, including information about 
diagnosis, treatment, conse- 
quences, new research and serv- 
ices available for its victims and 
their families What do you think 
about this idea (the new Center)'? 
Do you think that it is a good idea or 
nof? Wh/? 

2. What kind of information would be 
most valuable for you. if you had to 
cope with someone who had 
Alzheimer's Disease (diagnostic, 
treatmeni. medical referral, social 
services referral, support groups, 
respite care— probe for meaning of 
respite care)'? 

3. What advice would you have for the 
Institute as they plan for this new 
Center'? 



I will make sure that they know about 
your opinions 

Thank you very much for your 
contributions 
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Gatekeeper/Professional Review Questionnaire 

Source: Office of Disease Prevention and Health Promotion. USPHS 



INSTRUCTIONS: Please review the attached skill Your Name. 

sheet and then answer the questions below Return 
the completed questionnaires and skill sheets in the 
enclosed stamped envelope within the next three 

days Skill Sheet Reviewed: 



1. What would you say are the mam messages being communicated in this skill sheets 



2. How important do you think th.oe messages are for older people^ 

Very important 

Somewhat important 

Not particularly important 



3. In your professional opinion, are the recommendations made here appropriate for older oeop\e"> 

Yes No 

Why? 



4. !s there anything in the skill sheet that you found vague, confusing or unclear'? 
Yes No 

If yes, whaf? (Please note in space below or directly on skill sheet.) 



5. In your opinion, is there anything in the skill sheet that is inaccurate or misleading'? 
Yes No 

If yes, what? (Please note below or directly on skill sheet ) 



6. Do you feel there are any important points relating to this topic that have been omitted'? 

Yes No 

If yes, what are they'? 



7. Is there anything in the skill sheet you think older people might find offensive'? 

Yes No 

If yes, whaf? 



8- Which of the following phrases best describes how you feel about the visuals (illustrations, photos, etcj 
used in the skill sheef? (Circle one from each pair) 

a. Visuals effectively support a Visuals are relevant to the target 
the message audience 

b. Visuals do no* effectively b Visuals are not relevant to the 
support the message target audience 
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9. What, if anything, do you particularly like about the skill sheet? 



10. What, if anything, do you particularly dislike about the skill sheet? 



11. Which of the following groups (audiences) do you feel this skill sheet is addressing^ (Check one) 
All adults 

All adults but especially older people 

Only older people 



12. How responsive do you feel older people will be to the message contained in the skill sheel^ 

Very responsive 

Somewhat responsive 

Not responsive 

Why do you %el this way? 



13. Which of the following phrases best describes your 
are likely to view //? (Circle one in each group) 

a. Very informative 

b. Somewhat informative 

c. Not very informative 

a Very easy to understand 
b Fairly easy to understand 
c Not so easy to understand 

a. Very encouraging in tone 

b. Somewhat encouraging in tone 

c. Not particularly encouraging in tone 



opinion of this skill sheet in terms of how older people 

a Very realistic 

b Somewhat realistic 

c. Not particularly realistic 

a Very appealing 

b Somewhat appealing 

c Not particularly appealing 

a Very motivating 

b Somewhat motivating 

c Not particularly motivating 



14. How could the message in this skill sheet be improved? 



15. Based on what you know about the older target audience, are there any other comments you want to make 
about this skill sheet? 



THANK YOU FOR YOUR TIME PLEASE RETURN THE QUESTIONNAIRE AND SKlLL SHEET TO 

IN THE ENCLOSED, SELF-ADDRESSED. STAMPED ENVELOPE. 
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Standard PSA Pretest Questions 



1. Main Idea Communication/Comprehension 

What was the mam idea this message was trying to get across to you? 



What does this message ask you to do*^ 



\\lha\ action, if any, is the message recommending that people take'^ 



In your opinion, was there anything in the message that was confusing'^ 



Which of these phrases best describes the message'^ 

Easy to understand 

Hard to understand 

2. Likes/Dislikes 

In your opinion, was there anything in particular that was worth remembering about the message'^ 



What, if anything, did you particularly like about the message'^ 



Was there anything in the message that you particularly disliked or the.t bothered you'^ If yes, whaf^ 



3. Beiievability 

In your opinion, was there anything in the message that was hard to believe'^ If yes, whaf^ 



Which of these words or phrases best describes how you feel about the message'^ 

Believable 

Not believable 
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4. Personal Relevance/Interest 

In your opinion, what type of person was this message talking to 

Was It talking to . . 

Someone like me 

Someone else, not me 

Was it talking to . . 
All people 

All people but especially (the target audience) 

Only (the target audience) 

Which of these words or phrases best describes how you feel about the message'^ 

Interesting 

Not interesting 

Informative 

Not informative 

Did you learn anything new about (health subject) from the message'^ If yes, whaf? 



5. Other Target Audience Reactions 

Target audience reactions to messages can be assessed JSing pairs o^ words or phrases or using a 5-point 
scale. The following is an example of how this is done. 

Listed below are several pairs of words or phrases with the numbers 1 to 5 between them I'd like you to 
indicate which number best describes how you feel about the message The higher the nu iber, the more 
you think the phrase on the right describes it. The lower the number, the more you think the phrase on the 
left describes it. You could also pick any number in between Now let's go through each set of words Please 
tell me which number best describes your reaction to the message. 

Too Short 1 2 3 4 5 Too Long 
Discouraging 1 2 3 4 5 Encouraging 
Comforting 1 2 3 4 5 Alarmmg 
Well Done 12 3 4 5 Poorly Done 
Not Informative 1 2 3 4 5 Informative 
Is there anything in the message that would bother or offend people you know'? 



6. Impressions of Announcer 

Please select the one answer from each pair of phrases which describes your teehng^ ab':^'jt the announcer 

Believable 

Not believable 

Appropriate to the message 

Not appropriate to the message 

Gets the message across 

Doesn't get the message across 
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H(jiii^^^^ for PSAs 



Summary 

This television PSA pretesting nnetho- 
dology is a modified version of the 
theater-testing techniques used by 
connmercial advertisers. Here is how it 
works: 

Individuals typical of your PSA's 
target audience are invited to a con- 
veniently located nneeting roonn. The 
room should be set up for screening a 
television program. Participants should 
not be told the real purpose of this 
gathering: instead, that their reactions 
to a television program are being 
sought. 

At the session, participants watch a 
television program. The program can 
be any non-health, entertaining 15-30 
minute (approximately) videotape. The 
videotape is interrupted about halfway 
through by a sequence of four com- 
mercials. Your test PSA should be 
inserted behveen the second and third 
commercials (as indicated in the 
diagram below). 

At the end of the program, the 
meeting host (you or one of your col- 
leagues) asks participants to write in 
their answers to questions designed to 
gauge their reactions to the program. 
Then the host asks about the adver- 
tisements participants saw. Then, your 
test PSA is played again, by itself, and 
the host asks participants specific 
questions about their reactions. 
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Preparing for the Pretest 

Preparing involves: 

1. producing the PSA in rough form 

2. planning the pretest 

3. preparing the questionnaire 

4. recruiting respondents. 

St^p 1: Producln9 th« PSA In Rough 
Fonn 

The three most commonly produced 
forms of rough messages are "ani- 
matics," "photomatics," and "rough 
live action." If line drawings are used, 
the •'ough PSA is called an "animatic"; 
if photographs are used, the rough 
PSA is called a "photomatic." A 
"rough live action" message is 
produced by filming or videotaping an 
actual run-through of the script — using 
simplified sets, live actors, easily ac- 
cessible locations, or simulated back- 
grounds (e.g., rear screen projection of 
the set). In each case, the videotape 
includes an -audio delivery of the 
script. 

The storyboard serves as the blue- 
print for producing animatics. photo- 
nr^atics or rough live action messages. 
When producing animatics or photo- 
matics, the illustrations or photographs 
must be large enough and clear 
enough for videotaping. Each illus- 
tration or photograph should be at 
least 9" X 12" so that the camera can 
capture sufficient detail. Larger sizes 
also permit camera movement (e.g., 
moving left to right) within the frame to 
create a sense of motion or action. 

If you produce a photomatic, using 
slides projected on a screen allows 
you to create whatever size scene you 
want. 



Program 
(1st halO 



Program 
(2r>d halO 



>bur 
Test PSAs 



Control ^ 
Comnoercials 
1 &2 



^ ^bur Control ^ 
lest PSA Commercials 
3&4 




The more the rough message 
approximates final production quality, 
the more likely the pretest results will 
predict audience response accurately 
—the illustrations should be realistic, 
the characters should look like the 
characters you plan to use in the final 
spot, facial expressions should reflect 
the mood and tone of the script and 
the settings should be sketched in 
detail. If photographs are used, the 
pictures of people and places should 
be clear and should resemble those to 
be used in the final. Pay attention to 
the setting, wardrobe, props, the 
camera angle and perspective. Review 
the processed photographs and select 
the clearest ones for producing the 
rough message. 

The video portion of an animatic or 
a photomatic is produced by video- 
taping each scene, frame by frame. 
Simulate motion by moving the cam- 
era in or out ("zooming"), left to right 
("panning"), or up and down. Record 
the au' * portion of the rough mes- 
sage arid then edit to exactly the right 
length. 

Finally edit the videotape "to time" 
(30 or 60 seconds) using the sound- 
track as a guide "ihe video and audio 
tracks are then "mixed" together to 
produce (he rough message. 
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Summary— Producing tha Rough PSA 



A. Animatics and Photomatics 

1. Develop a stofytx>ard. 

2. For photomatics, arrange talent. Ir tion setting and props. 

3. Record video portion frame by fr< . 

4. Record audio portion and edit it lo time." 

5. Edit videotaped footage "to time" 

6. Mix audio and video tracks. 

a Rough Uvt Action 

1. Develop a storytx>a''d. 

2. Arrange talent, localion, setti.igs and props. 

3. Rehearse the spot. 

4. Videotape (with sound) the rough live action. 



If you decide to videotape live 
action instead of pictures, use nonpro- 
fessional actors (e.g., friends or co- 
workers) to enact the script in a setting 
that closely approximates that to be 
used in final production. The visuals 
and the sound should be recorded at 
the same time. To minimize the num 
ber of times the live action must be 
taped, the actors si^ould rehearse and 
the production crew should be briefed. 
A detailed production plan (discussed 
below) will guide the crew. 

How can t Iceep rough message 
production costs down? 

Animatics, photomatics and rough live 
action PSAs can be producea wiih the 
help of a professional production 
company, your agency's audiovisual 
staff, a local television station or a 
local college or university The pro- 
fessional production ccmpcny will 
probably be the most expef^^Jve 
option. The latter three options car. be 
less expensive. For example, a local 
television station may donate its ser- 
vices for producing your spot Or, a 
television production instructor may 
assist you at no charge (except for the 
cost of the videotape) by making 
production of your rough message a 
ctudent assignment. 



Tnere are several wa^ > J -"^ • 
control the costs and tht p. xiuction 
quality of your rough PSA. 

1. Create a detailed production plan 
including: 

o What scenes will appear on the 

screen and in what order^ 
o How long will each scene be on 

the screen'? 
o What camera movements will be 

needed in each scene? 
o how w:II scenes be edited 

toget'irr (e.g., fades, dissolves or 

direct cuts)? 
n ^Vhat p':)rtions of ihe soundtrack 

go with eac'i scene'? 

2 Send your production plan in 
advance to the person who will be 
helping you, and have extra copies 
on hand at the production session 

3. ivlake sure ycur script and produc- 
tion plan for the rough message are 
complete and timed in advance. 

4. Pierecord the audio track before the 
visuals are shot 

5. Make sure your illustrations, photo- 
graphs and/or slides are m the right 
order when you arrive for the pro- 
duction session. 

6 Finally, remember that you are 
producing a 'nugh message. Save 
"perfection' -or final production' 



Rough message production costs 
rnay be reduced further by using 
amateur talent, friends or co-wckers 
who can do a respectable job 'eoord- 
ing the PSA script. These same people 
may be used for photographs or in a 
rough live ectic^ sp:! Shooting the 
visuals and recording the soui .d^^^^k 
on the same oay al'io may save time 
and money Finally consider contact- 
ing a local radio stat'on for recording 
the soundtrack — the station may 
orovide their facilities free of charge or 
at a lower rate than a recording studio 

Animatics may be less expensive 
and quicker to produce than photo- 
matics. Since they are composed of 
artists' renditions. While you mv^v h^ve 
to buy the artwork, you elimir;qte costs 
for taLnt (actors), location set^ ng (le 
obtaining clearance to rent or use 
property), props and travel However, 
drawifigs may not communicate the 
realities or subtleties of the visual 
portion of the message as //ell as 
photf^r-Hphs or rough nve artion 
These drawbacks can be minimized by 
I 'Sing a g od illustrate, and a good, 
clear soundtiack. 

Photomatics rnay be mo's expen- 
sive than animatics, depending open 
the cost of the fhotography The 
availability of people who can r< 
sent the characters. *he acces^"iibiJity 
of an appropriate location, and thj rate 
your ohotographer charges are xr^e 
major factors that affect costs. These 
costi oan be kept down by workmp 
with people who are readily avaiJ':*^le 
and by arranging all the details ir 
advance The major advir.tage of 
producing a photomatic over tn 
animatic is thai you can proou« a a 
more realistic ana believable rer;dition 
of the final message Compare the 
costs of an illustrator versus a 
photographer and factor in the 
logistics involved when deciding 
whether you should produce an 
animatic or a pnotomatic 
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A rough live action message can be 
the least expensive way to produce 
your message in rough form, if: 

o you are using amateur talent who 
can do a respectable run-through of 
the script; 

0 your script can be produced in a 
readily accessible location th^t does 
not require a lot of set-up time; and 

□ your production crew can videotape 
the run-through with a minimum 
number of "takes." 

We recommend that you videotape 
rather than film the rough message for 
pretesting becaus.. editing and pro- 
cessing are ucjally less expensive 
Home or studio video equipment can 
be used to produce any of the three 
rough message forms. 

What do I do once I Ve produced 
the rough message? 

You need to locate a dramatic or non- 
health documentary videotape to serve 
as the program to be "reviewed" by 



respondents. You may find a suitable 
program through your library, a free- 
loan service (ask your librarian) or you 
may tape a television program (try PBS 
for a program your respondents may 
not have seen). You'll also need to tape 
four non-health conMTiercials from 
television. 

The PSA pretesting program is 
interrupted about halfway through by a 
series of commercials. Insert two 
commercials, your test PSA. then two 
more commercials. Also edit your PSA 
onto the videotape about five seconds 
after the end of the test program (see 
the diagram on page 101). 

St«p 2: Planninf tht Prtt«st 

The purpose o:' the pretest is to deter- 
mine what improvements, if any, 
should be made before final produc- 
tion. To guide you in preparing the 
pretest questionnaire and in analyzing 
the results, write a clear statement of 
your message's objectives, the target 
audience and the main point or points 
your PSA is trying to get acrc-^s. In 




addition, identify any special concerns 
you have at)Out the PSA. 

PSA producers often have certain 
concerns about target audience reac- 



Rough PSA Type Advantages Disadvantages 

Animatic □ Lbss expensive than others o May not approximate script 

□ Fewer logistical arrangements □ Needs artist who can render clear 

□ No props required ''"^ drawings 

□ No location settings 

□ Easiest to produce 



Photomatic □ More realistic than animatic □ More costly than animatic 

□ Can be inexpensive and easy to □ More logistical arrangements than 

produce if location, props, and animatic 
actors are available 



Rough Live Action □ More realistic than animatic or 

photomatic 

□ Can be inexpensive if location, 
props, and actors are readily 
available 



□ Many logistical arrangements, 
rehearsals required 

□ May be more difficult to make 
changes 



Write out message objectives, intended target audience, and main 
points. 

identify special concems, eg., technical words, believability, music. 
Consider each message element with audience recall, comprehension 
or other reactions in mind. Decide which concerns take priority. 



tions to their messages. For example, 
technical words or phrases could be 
confusing. Or, the characters in the 
spot could be perceived as unbeliev- 
able or inappropriate. Or the music 
could be a problem. 

Now is the time to consider each 
element in the message that may 
affect audience recall, comprehension 
or other reactions, and to determine 
which of these concerns should be 
addressed in pretesting. 

What are your specific objectives in 
testing this television PSA? 

Fill out the pretest planning form on 
page 113 for the PSA you will be 
*w';>ting. Consult this form as you 
develop your questionnaire 

Stsp 3: Dmioping the Questionmiirt 

To gather useful information from the 
pretest, the questionnaire must be 
carefully constructed. It will contain 
four parts: 

1. Program Questions — regarding 
audience reactions to the television 
program to be presented at the test 
session. All you need to do is copy the 
Fiart I questions from the Sample 
Questionnaire on page 115. Use it as 
page 1 of your questionnaire. 

2. Recall and Main Idea 
Communication— \h;ee standard 
questions to assess your PSA's ability 
to attract attention and to convey its 
main point will appear on page 2 of 
your questionnaire Again, simply copy 
the Part II questions from the Sample 
Questionnaire. 

3. A'jdience Reactions — several 
standard audience reacUon questions, 
and questions you wil' develop 
specifically for your test message 

4. Demographics — questions to 
record the characteristics of the 
participants (e.g,. their sex. age, level 
of education, Iiealth status, etc). 

Each part of the questionnaire 
should be separated by a piece of 
paper telling respondents not to go 
further until they are instructed to do 
so 



Summary— Manning tiie Pretest 



1. 

2. 
3. 



How does the questionnaire work? 

After participants have viewed the 
television prograrri and the commercial 
sequence, the meeting host asks the 
questions about the television program 
on page 1 of the questionnaire These 
questions are asked because partici- 
pants were told that the purpose of the 
test session is to get their reactions to 
a new television program. (In fact, 
page 1 can be discarded after the 
pretest session.) 

After giving participants time to 
write their answers on page 1. the host 
reads aloud standard pretesting ques- 
tions (Part II) and gives participants 
time to fill in their responses. 

The first two questions measure 
"audience recall" or the ability of your 
message to attract attention. The third 
question is designed to measure the 
extent to which your message commu- 
nicated its main idea or point. It is not 
enough for an announcement to be 
remembered; it must also communi- 
cate the key information to its 
audience. 

The idea! results would be for 
everyone to recall seeing your PSA 
and to remember its main point. 
(Although the recall and main idea 
communication questions will yield 
information on the other messages in 
the commercial sequence, you will 
disregard that information when 
analyzing the pretest results.) 

Once participants have completed 
page 2. the host should finish playing 
the videotape to show your PSA a 
second time. The host should then ask 
the main idea question again. (See 
Part III.) Usually more respondents will 



.to 



be able to descnbe the main idea 
following their second exposure, since 
the message is shown by itself and. 
therefore, is not competing with other 
message^.. 

In addition to completing the main 
idea question after their second expo- 
sure to the message, participants 
should be asked a series of questions 
to gauge their reactions to your 
message. It is up to you to decide 
which questions from the Sample 
Questionnaire you need to ask. 

Why should you use the standard 
questions? 

The standard recall and reaction 
questions should be incorporated into 
the pretest for two reasons. First, the 
questions address the most important 
measures of a message's potential 
effectiveness — i.e., whether it attracts 
audience attention and whether it 
conveys its main point. 

Also, the use of standard questions 
allows comparison, because these 
questions were used in many pretests 
of television PSAs conducted by the 
Hoalth Message Testing Service. The 
•esults of these previous pretests can 
be used as a guide to assess how well 
your message communicates. (See 
page 110 for comparisons.) 

Why do i t\eh6 to develop additional 
questions? 

You should develop questions that 
address your special concerns about 
your PSA. For example, suppose your 
message asks viewers to call a toll- 
free number for more information. You 
may want to include a question that 
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Use of music (with or without lyrics) 

Use of famous spokesperson 

Use of telephone number 

Use of mailing address 

Request for a particular action 

instructions tor performing a specific health behavior 

Presentation of technical or medical information 

Presentation of new information 

Promotion of a sponsoring organization or event 

Characters intended to be typical of the target audience 

Use of a voice-over announcer 

Presentation of controversial or unpleasant information 



asks, "What action, if any, is the 
message recommending that people 
take?" A related question may be. "Did 
the telephc'"'^ number appear on the 
screen long enough for you to write it 
down?" 

How do I know what to ask? 

It IS best to develop one or more 
questions addressing each message 
characteristic in your PSA. The chart 
below lists various message character- 
istics which are commonly found in 
PSAs. Check those characteristics 
which apply to your test message. 
Examples of questions for each 
message characteristic listed above 
are in Part III of the sample question- 
naire. These questions are just 
examples— adapt the questions to 
your needs. Remember that the 
objective of pretesting is to uncover 
any problems with your PSA prior to 
final production. Any serious concerns 
you have about your message should 
be explored in pretesting. 

How do I get the questionnaire 
ready for the pretest? 

Once you have written your question- 
naire, you are ready to make copies for 
the pretest participants. You will need 
a cover page that instructs partici- 
pants not to open their questionnaires 
until they are told to by the meeting 



host. Place a cover sheet between 
each part of the questionnaire that 
instructs participants not to go further 
until they are told to do so by the 
meeting host. 

Make extra copies of the question- 
naire in case rwre people attend the 
pretesting session than you expect. 
For example, if you are expecting 50 
participants, make 60 to 65 copies of 
the questionnaire. 

St«p 4: Recruiting Participants for tha 
Pratast 

Before recruiting respondents, identify 
the torget audience for your message. 
These are the people you will want to 
recruit. 

How many participants should be 
recruited? 

This pretesting methodoiogy is de- 
signed to provide qualitative, diagnos- 
tic information Therefore, there is no 
statistical procedure for determining 
the sample size The sample size 
should be large enough, however, to 
give you confidence that you have 
sampled a range of opinions. A 
reasonable and adequate sample size 
IS 50 participants typical of your target 
audience. 

If participants are to reflect a gen- 
era! audience, the sample should 
include a mix of men and women. 



representing a range in age, education 
and income levels. To achieve this 
variation, we recommend tfiat parti- 
cipants be recruited from at least two 
different sources. If the message is 
intended for both a general audience 
and a specific target group (e.g., 
blacks), you should recruit two 
samples of 50 participants each. 

To assure that 50 participants 
attend, you should over-recruit. That is. 
for every sample of 50 participants, 
recruit about 75 people. This is 
necessary because there are always 
participants who do not show up. 
Calling and reminding people the day 
before the test will help the attendance 
rate and will help you predict any 
problems w th low attendance. If you 
choose a "captive" audience (e.g.. 
seniors enrolled in a senior citizens 
center activities program, children in 
school or PTA members at their 
meeting), you may not need to recruit 
individuals, but just make arrange- 
ments with the organization. 

An important consideration related 
to recruitment is the type of facility 
and the viewing arrangements you will 
be using. If you recruit all 50 partici- 
pants to one test session, you will 
need at least two television monitors 
so that everyone can see the video- 
tape. It may be necessary to set up 
two or more test sessions for smaller 
facilities. Be sure to choose a location 
that is convenient for your ta get 
audience. 

How are participants recruited? 

One technique that has been used 
CLJCcessfully to recruit participants for 
P''etesting health messages is to seek 
the cooperation of community groups 
such as civic, social o: religious 
organizations whoso members are 
typical of your target audience. 
Schools, health clinics and work sites 
may also be contacted for assistance 
in recruiting participants. These 
organizations usually have member- 
ship lists which can be used for 
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recruiting purposes. This is especially 
useful if you need to recruit a specific 
target audience and you can identify 
an organization whose members 
match your needs. Also, members are 
more likely to participate as a gesture 
of their affiliation or for the benefit of 
their organization. A contribution of 
$25 to $50 IS an attractive incentive for 
gaming a community group's coopera- 
tion. In addition, you may want to 
obtain a token gift from a local mer- 
chant to raffle as an extra incentive. Or 
you may provide small individual gifts 
or offer refreshments at the pretesting 
session. 

The following list contains the 
examples of community groups with 
local affiliates throughout the country 
that you might consider contacting for 
recruiting participants. 



Women 

YWCA 

B'nai B'rith Women 
Junior League 
jjeague of V\fomen Voters 
Parent Teacher Associations 
National Council of Catholic Women 
Church or syrjagogue groups 
Women's business groups 
Work sites or unions 
Senior citizen centers 

Men 

YMCA 

American Legion 

Elks 

Eagles 

Jaycees 

Knights of Columbus 
Lions Club 

Loyal Order of Moose 
Rotary Club 

Veterans of Foreign Wars 
Local business groups 
Work sites or unions 
Senior citizen centers 



Once you have identified potentially 
cooperative community groups, con- 
tact an official within the organization 
(e.g., the president or program direc- 
tor). While these initial contacts may 
be made by telephone, you should 
follow up your discussions with a 
formal written request for cooperation. 
The written request offers the opportu- 
nity to outline several impoilant points, 
including: 

a a description of your agency or 
organization 

□ a description of the PSA and its 
purpose 

□ details regarding the participants to 
be recruited 

□ an outline of the activities involved in 
the pretest 



Blacks 

National Council of Negro Women 
National Urban League 
Parent Teacher Associations 
National Association for the 

Advancement of C/Olored People 
Church groups 
Senior citizen centers 
Work sites or unions 

Teenagers 

Schools (auditorium), or test as a 
part of a class 



:3 incentives you are offering the 
organization and/or the participants 

□ a detailed explanation why the 
official should not reveal the true 
nature of the pretest to participants 

Once you have obtained cooper- 
ation, decide how participants will be 
recruited. One possibility is to conduct 
the pretest as part of the group's regu- 
larly scheduled meeting. In this case, 
all members who attend the meeting 
will participate in the pretest. The 
advantages of this approach are: 

□ relatively little extra effort is 
expended in recruiting participants 

a participants may not have to be told 
about the pretest until they arrive at 
the meeting, thus minimizing the 
possibility of their learning about the 
true nature of the pretest 

□ their regular meeting facility, a 
convenient and familiar location, 
may be used to conduct the pretest. 

If some meeting participants do not 
meet your screening requirements, 
their responses can be discarded later. 

The disadvantages of this approach 
are: 

□ there is little control over the number 
of people who will come and over 
the composition of the group 

□ because the pretesting session lasts 
approximately one hour, it may be 
difficult to place it on the agenda of 
a regularly scheduled meeting 

□ a number of organizations arrange 
their programs mary months ahead 
of time (it may be difficult to sched- 
ule the pretest within a reasonable 
time frame). 

An alternative is recruiting the 
group's members to a special meeting. 
In ih'S instance, members may need to 
be contacted and asked to participate 
either by someone from your agency 
or by someone from the community 
group. In either case, the members 
should be contacted in the name of or 
on behalf of their own group. 



Community Oraanfaeations to Contact For Recruiting Participants 
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□ You need at least 50 participants 

□ For a general audience, recruit a nnix of men and wonnen, with a range in 
age, education and inconne levels. 

□ Over-recruit — 75 participants for every 50 desired. 

□ For a nnessage interxled for both a general audience and a target group, 
recruit two sannples. 

□ Plan tv/o or nnore test sessions if necessary. 

□ Rejruit target audiences through civic, social or religious organizations; 
or through schools, clinics or work sites. 



Sununary-Rtquirtmtfrts for Rtcrutting 



The major advantages of recruiting 
participants independently of sched- 
uled meetings are: 

□ the opportunity to screen partici- 
pants on relevant charactenstics 
and to elinninate, in advance of the 
pretest, health professionals or other 
respondents who should not 
participate 

□ since the entire "meeting" will be 
devoted to the pretest, participant 
fatigue is reduced. 

Disadvantages of recruiting partici- 
pants for pretesting apart from a 
scheduled meeting include: 

□ the amount of time and energy 
involved in contacting and reminding 
participants of the pretest 

□ the necessity of locating special 
facilities for the pretest (if the regular 
meeting facilities are not available). 

If participants are recruited by your 
agency, you will have more control 
over what the participants are told 
about the pretest, and you will be able 
to screen potential participants to 
make certain that they fit any special 
audience characteristics. However, 
recruiting participants takes a signif- 
icant amount of time, and it is possible 
that people would be more likely to 
participate if asked by someone whom 
they know. 



If participants are contacted by the 
member organization, it is essential 
that you provide the recruiter with 
detailed instructions for carrying out 
the task. These instructions must 
include a written description of the 
"disguised" pretest, which should be 
read to potential participants verbatim, 
and a questionnaire to screen partici- 
pants on relevant characteristics (e.g., 
how many children they have, whether 
they have high blood pressure). A set 
of questions that can be used for 
screening participants and for 
identifying the characteristics of 
participants in the test session is 
included in Part IV of the Sample 
Questionnaire. 

One way to disguise the purpose nf 
the pretest is to tell participants they 
are being invited to screen a new 
television program and that their 
reactions are being sought by the 
producers. No reference should be 
made to the commercials or to the 
name of your agency since this could 
bias their response and sensitize 
participants to the nature of the 
research before they arrive at the 
pretest session 



In pretesting health PSAs, health 
professionals should be excluded from 
participating. In addition, people who 
are familiar with your agency's 
program should not participate in the 
pretest, since their responses are likely 
to be biased. 

Conducting the Pretest 

Prior to the pretest session, go over 
the checklist below to be sure that 
your session will go as smoothly as 
possible A rehearsal of the pretesting 
session at your own office is an 
excellent way to avoid anv problems. 

How do I conduct the ptotast 
session? 

The procedures to follow during the 
pretest session are relatively simple. 
The keys to a successful testing 
session are: 

□ having everything organized and in 
operating order before the session 

□ being friendly and courteous to 
participants from the moment they 
arrive until they leave (remember to 
say "thank you") 

□ keeping calm and cool headed 
throughout the session 

□ anticipating problems in advance 

The test session should take no 
more than an hour and 15 minutes if 
you are organized and well prepared. 
(See the end of this Appendix for a 
sample script for hosting the session ) 
Encourage participants to take a seat 
as they arrive, close the doors no later 
than 10 minutes after the scheduled 
starting tiir.e. When everyone is 
seated, introduce yourself by name 
only (assuming you are the host;. Do 
not tell participants the name of your 
organ'zation during the session be- 
cause *his might bias their responses 
to yobr test PSA. Thank participants for 
coming and assure them that the 
evening should be enjoyable— one in 
which they will have a chance to give 
their views to the producers of tele- 
vision program material. Discourage 
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Summiry-4ttps for itocniiting Participants 



1. Identify community groups representative of your target audience 

2. Contact official in writing. 

3. Decide whether pretest will be held at regular meeting or at special 
meeting. 

4. If pretest will be held at special meeting, recruit participants individually, 
preferably through your own agency 



Is the pretesting videotape ready for use^? Are the video and audio 
portions clear? 

Is your videotape equipment — recorder and television monitors — 
functioning properly? 

Is the meeting room or other facility set upP Is the room reserved for 
you? Are there enough chairs? Are extra chairs available in case nnore 
people show up than you expect? Do you need another TV monitor so 
that everyone will be able to see the program? Is the heat or air 
conditioning working properly? Do you know where the light switches 
are? If a microphone is needed, is it set up and functioning properly? 
Have you made all the necessary staffing arrangements? Are your 
assistants definitely coming to the session? Do they have transportation 
and do they have directions for getting there? 
Have you made enough copies of your pretest questionnaire? Is each 
questionnaire complete (with no pages missing)? Do you have pencils 
for participants? Will they need clipboards or pads? 
Has participant recruitment taken place as scheduled? Did you call and 
remind them to attend? Do they have transportation and directions? 
Has the host rehearsed? 



Chackiist-Planning tha Pratatt 



1. 
2. 
3. 



4. 



5. 



6. 
7. 



participants from talking to one 
another during the session. Tell them 
you are interested in their own par- 
ticular views and that there are no 
right or wrong answers. Also, 
encourage them to write their answers 
clearly in the space provided. 

After your introductory remarks, 
have your assistants hand out the 
questionnaires, pencils and clipboards 
(if needed). Turn on the video recorder 
and monitor to begin the test session. 

Be attentive and watch for any 
problerr.b with the sound or picture on 



the monitor Be sure that the equip- 
ment IS functioning properly through- 
out the program. 

Be prepared to stop the recorder 
when the television program has 
ended. Introduce the questions, 
thanking the group for their help thus 
far. Tell them to open their question- 
naires to the questions on the first 
page. Read each question and give 
respondents time to fill in their 
answers. 



When the participants are through 
with page 1, tell them that before they 
leave, you would like to gather their 
reactic is to the announcements that 
were sliown during the program. Have 
them turn to page 2 and instruct them 
in how to fill out the questions about 
the advertisements. When they have 
completed these questions, tell the 
group that you have been asked to 
obtain their reactions to one of the ads 
in particular — the public service 
announcement. 

Start the recorder Be sure there is 
not too much lead tape before the 
message starts, to avoid an awkward 
pause in the session's pace. After your 
PSA has been played, tell participants 
to turn to the next page of the ques- 
tionnaire and to write their answers to 
the remaining questions. Encourage 
them to answer every question and to 
avoid giving more than one answer, 
except when this option is indicated on 
the questionnaire. 

It is a good idea to circulate through 
the room to monitor progress and to 
be sure participants are not discussing 
their responses. Collect the question- 
naires as participants finish. 

Thank participants for their coopera- 
tion. If you have token gifts (e.g., 
pencils, cent-off coupons, half-price 
tickets or other tavors from local 
merchants), distribute them to partici- 
pants as they leave. Otherwise, 
mention that you hope the group 
enjoys the donation you have made to 
their organization. 

Congratulations on conducting a 
successful pretest! 
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Be well prepared and organized. 

Introduce yourself by nanne only, and express your interest in indivi- 
dual responses to television program material. 
Pass out questionnaires, pencils and clipboards. 
Play videotaped program, watching for any sound or picture problems. 
Read each question on page 1 of the questionnaire, and allow time for 
participants to fill in answers. 

Have respondents fill in page 2, following your instructions. 
Play the PSA. 

Have participants fill in remainder of questionnaire. 
Collect questionnaire. 
Give out gifts or favors. 

Say thanks and mention your donation to their group 



Summary— Conducting tho Prttost 



1. 
2. 

3. 
4. 
5. 

6. 

7. 

8. 

9. 
10. 
11. 



Analyzing the Pretest Results 

Analysis of the questionnaires involves 
two steps. First, tabulate or count how 
many participants gave each possible 
response to each question. Next, look 
for patterns in the responses and 
extract meaning from the numbers. 

How do I tabulate closed-ended 
questions? 

Closed-ended questions orce 
participants to select a response from 
several alternatives. A quick method 
for tabulating or counting the number 
of participants who selected each 
possible response to each question is 
to use a blank questionnaire: 

1. Take thp first questionnaire and 
record the answers to each closed- 
ended question by making a check 
nnark in the right hand column next 
to the appropriate response. 

2. Repeat this procedure for every 
questionnaire. 

3 Tally the total number of check 
marks and then calculate the 
percentage of participants who 
gave each response. 



How do I tabulate open-ended 
questions? 

Tabulating or counting responses to 
open-ended questions is more time- 
consuming. Open-ended questions 
allow participants to express 
themselves in their own words. For 
example: 

c What did the message say? 

□ W'^at did the message show^ 

□ What was the main idea the 
message was trymc, to got across? 

The easiest way to analyze these 
questions is to write each question at 
the top of a separate blank page (the 
"coding sheet"). Since participants are 
answering in their own words, the first 
step IS to group the responses to each 
question into categories. 

Group the responses to each ques- 
tion into three categories: correct, 
partially correct and incorrect. When 
categorizing the audience recall and 
mam idea responses, use the PSA 
script and your own statement of the 
message's main points to guide the 
analysis. For some open-ended ques- 
tions (e.g.. "What, if anything, did you 
like about the message?" or "What, if 
anything, did you dislike ?"), you may 
wish to classify the responses as 
favorable or unfavorable. 



How do I Interpret the pretest 
results? 

Once you have tabulated the re- 
sponses to each question, your inter- 
pretation should be guided by the 
objectives you stated for your mes- 
sage on your pretest planning form 

Guidelines for interpreting re- 
sponses to standard pretesting 
questions are listed on page 110. 
These guidelines are in the form of 
high, average, and low percentages (or 
scores) for certain standard pretesting 
questions. 

The ranges are based on the results 
of 63 television PSA pretests. They can 
be helpful in comparing your PSA 
pretest to previous pretest results. This 
comparison should not be the sole 
basis of your analysis. Rather, the 
ranges should serve as guidelines. 
They are designed to help you extract 
meaning from the percentages you 
compute for your test PSA. 

Ideally, your PSA pretest results will 
fall in the average or the high range, 
and not the low range. One or two low 
scores do not necessarily dictate 
revising the PSA. Prior to drawing any 
conclusions or mak.ng recommenda- 
tions, you should analyze the results 
for every question. 

This book cannot provide ranges or 
guidelines for the questions which 
relate specifically to your test PSA. 
Therefore, you will have to interpret the 
percentages on your own. Ask yourself 
how critical it is for the majority of 
participants to respond in a certain 
manner to each question. For example, 
if the test PSA communicates the main 
idea information through music, it is 
very important that a large majority 
report being able to understand the 
words to the music. Otherwise, the test 
PSA will not meet its communication 
objectives. On the other hand, if the 
music in the PSA serves as back- 
ground sound, then it is not as critical 
for the lyrics to be understood. 

It is important to examine how many 
responses to open-ended questions 
fall into each category (correct, 
partially correct or incorrect, favorable 
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HlghScortRangn 


Avifigt Scori Rangt 


Low Scot* Range 

70 


AttMitkm/IUcall 

(percent remembering seeing nnessage after 
one exposure) 


41 or higher 


30-40 


29 or lower 


Main Mm 

(percent rennembering main idea of message 
after ane exposure) 


36 or higher 


25-35 


24 or lower 


ffonn INNIMIIIU9I II in 

^|Ju[ v/c;i 11 11 ivilv/ailMy yco / 


76 or higher 


60-75 


59 or lower 


PtnomWy Itolmnt 

(percent indicating message is talking to 
someone like themselves) 


66 or higher 


50-65 


49 or lower 


Anything Confutinf 

(percent indicating "yes") 


9 or lower 


10-20 


21 or higher 




91 or higher 


75-90 


74 or lower 


WMDont 


66 or higher 


50-65 


49 or lower 


Convincing 


71 or higher 


55-70 


54 or lower 


Informathft 


76 or higher 


60-75 


59 or lower 


Mad^Kt Point 


91 or higher 


75-90 


74 or lower 


Intaratting 


66 or higher 


50-65 


49 or lower 


PlMMnt 


66 or higher 


50-65 


49 or lower 



or unfavorable). Ideally, the majority of 
responses will fall into the correct 
category. This pattern tends to confirm 
that the test PSA is meeting its objec- 
tives. On the other hand, if the majority 
of responses are categorized as par- 
tially correct, or incorrect, the PSA 
may not be meeting your communica- 
tion objectives. 

At this point, look at the overall 
results: 

° What was learned from the pretest'? 

a Did the PSa meet its communication 
objectives? 

a Wha: are the PSA's strengths? 
Weaknesses? 

a Did the PSA receive a favorable and 
appropriate audience reaction? 



□ Did any answers to any particular 
question stand out? 

o Should the PSA be revised? 

The answers to these questions can 
be found in your analysis However, it 
is no longer appropriate to look at 
each individual question 

Can I apply any statistical 
techniques to the data? 

For example, can the findings be 
analyzed in terms of males vs. 
females, or younger participants vs. 
older ones? The answer is yes — but 
only if you boost the sample size to 50 
participants from each group If you 
recruit only 50 participants in total, the 
sample will be too small to be 
reviewed by subgroup. Comparing 



findings between small subgroups 
could be misleading and may result in 
faulty conclusions. 

Do not try to make generalizations 
from the results of this pretest to PSA 
messages in general. Each PSA 
pretest you conduct will provide 
direction for making changes that 
might be necessary prior to final 
production and for improving the 
potential effectiveness of each 
particular message. The sample size 
and the methodology used do not 
allow for formulating definitive 
conclusions about what your 
message's impact will be after it is 
distributed. 
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Summary— Analyiing th« Prttett 



1. Tabulate how many participants gave each possible response to each 
question. 

a. Use check marks and percentages for closed-ended questions. 

b. Use categories ("correct/partially correct/incorrect" or "favorable/ 
unfavorable") and p^^rcentages for open-ended questions. 

2. Compare the results with HMTS pretest results, but use these as guide- 
lines only. 

3. Ux)k for patterns across questions. 

4. Compare resuits with your original objectives. 

5. Decide whether to recommend changing the PSA or to keep it 
essentially the same. 



Pretest Budget and Timetable 



Timetable* 




Budget** 






Out-of- 


Week 


Task 


Pocket Costs 


1 


Produce rough message and transfer to pretest videotape 


$1,000 


1-2 


Contact community organizations for recruiting participants 


100*** 


2 


Develop questionnaire 




2 


Begin recruiting participants 




3 


Identify and reserve facility for pretesting session 


50 


3 


Rent video equipment 


150 


4 


Pretest and revise questionnaire 




5 


Copy questionnaires 


20 


5 


Conduct pretesting session(s) 




6 


Analyze results and write report 




7 


Make script changes based on pretesting 





$1,320 



*The timetable presented here assumes that a health professional will have the assistance of a secretary and that the connmunity 
organization is handling participant recruitment The variable most likely to affect your pretest timetdDle is the amount of time needed 
to recruit participants 

**The out-of-pocket costs are estimates and do not include the cost of your time working on the pretest Costs will vary depending 
upun your rough message production costs, the number of community organizations you work with, whether or not you need to rent a 
facility or equipment, and your internal resources 

***Reflects a donation of $2 per participant recruited for the pretest 



ERLC 



Pretest Planning Form 

1. What is the communication objective of my PSA? 



2. Who is the target audience? 



3. What is the main or overall point I am trying to communicate to my audience with this spot? 



4, What other important points do I want the viewer to remember from this PSA? (List these in order of 
importance below.) 



5 What special concerns do I have about this PSA (e.g., reactions to the characters, music, technical words 
setting, etc.)? 
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Sample PSA Pretest Questionnaire 



Parti 

Thank you for watching this program. One of the reasons it was shown tonight was to get your reaction to it— to 
see what parts you liked and what parts you didn't like. 

1. Was there any part of the program that you especially liked? 



2. Was there anything about the program that you disliked? 



3. Please indicate your overall reaction to the program by circling or.e of the phrases below: 

a. A great program, would like to see it again 

b. A pretty good program 

c. Just so-so. like a million others 

d. Another bad program 

4, Would you recommend the program to your friends? Why or why not? 



Part 11 

For each commercial that you remember seeing, please write down what the message said and what the 
message showed on the dotted lines Write down the main idea each message was trying to get across on the 
solid lines. 

a) What did the message say? What was the main idea each message was trying to 

b) What did the message show? get across? 

1. a) . ^ 

b) ~ 



2. a) 
b) 



115 



ERLC 



3. a) 
b) 



4. a) 
b) 

5. a) 
b) 



Part Hi 

Listed below are the standard reaction questions we recommend you include in your questionnaire 

1. This evening you saw a public service announcement. Now that you have seen the message twice, please 
tell us what you think was the m?'n idea of the message 



2. In your opinion, was there anything in particular that was worth remembering about the message? 

a. Yes 

b. No 

2a. If yes, what was worth remembering? 



3. In your opinion, what type of person was this message talking to? 

a. Someone like me 

b. Someone else, not me 

3a. If someone else, why? 



4. In your opinion, was there anything in the message that was confusing or hard to understand? 

a. Yes 
b No 

4a. If yes. what was confusing? 
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5 We would now like you to describe thf )ublic service announcement. From each pair of words or phrases, 
please circle the one which you feel best describes the message. 

a. Believable a. Made its point 

b. Not believable b. Didn't make its point 

a. Well done a. Interesting 

b. Not well done b. Not interest!'-': 

a. Convincing a. Pleasant 

b. Not convincing b. Not pleasant 

a. Informative 

b. Not informative 

Following are sample questions for each message characteristic discussed in Chapter 2, Feel free to adapt 
these questions to your needs, or develop your own questions. Make sure your pretest questionnaire covers all 
aspects of your PSA. 

Use of Music: 

1 Please circle the one answer from each pair which better describes your feelings about the music in the 
message. 

a. Appropriate to the message 

b. Not appropriate to the message 

a. Effective in getting the message across 

b. Not effective in getting the message across 

a. Could understand the words to the music 

b. Could not understand the words to the music 

2. Overall, how would you describe the music in the announcement? 
a. The music fit the message 
D. The music did not fit the message 

c. I don't remember the music 

Use of Famous Spokesperson: 

1 Which of the following best describes (name of spokesperson), the announcer in the 

public service message? 

a. Singer 

b. Actor 

^ Comedian 

d. Athlete 

e. Don't know 

2 Please circle the one answer from each pair of phrases which better describes your feelings about the 
announcer, (name of spokesperson). 

a. Believable 

b. Not believable 

a. Appropriate to the message 

b. Not appropriate to the message 

a Gets the message across 

b. Does not get the message across 
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Use of Telephone Number or Address: 

1. The phone number (or address) was on the screen long enough for me .emember it oi write i 

a. Agree 

b. Disagree 

c. Neither agree nor disagree 

Request for a Particular Action: 

1. What did the announcement ask you to rJo? 



Instructions for Performing a Specific Health Behavior u ♦ ♦u 

1 Please circle one answer from each pair of phrases which better describes your feelings about the 
' instructions regarding (fi" in behavior) in the announcc^Tient. 

a. Clear and easy to understand 

b. Confusing, hard to understand 

a I would be able to perform (fill in behavior) after seeing this announcement. 

b' I would not be able to perform (fill in oehavior) af^.er seeing t.iis 

announcer it. 

Presentation of Technical or Medical Information: . 

1. The aniiouncement presented technical (or medical) information. Please select one answer from each pair of 
pnrases which better describes your feelings about the information. 

a. The announcement did a good job of presenting technical information. 

b. The announcenrient did a poor job of presenting technical information. 

a. I understood all the terms in the announcement. 

b. I had difficulty understanding the terms in the announcement. 

Presentation of New Infonnation: 

1. How much, if any, of the information in the announcement was new to you? 

a. All of it 

b. Most of it 

c. Some of it 

d. None of it 

2. Overall, how useful was the information in the announcement to you? 

a. Very useful 

b. Somewhat useful 

c. Not very useful 

d. Not usef'Jl at all 

e. Dcrri Know/not sure 
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Promotion of a Sponsoring Organization: 

1 From among the following choices, please indicate the organization which sponsored this message. (Include 
your organization and fill in appropriate alternatives.) 

a. CARE 

b. Baylor College of Medicine 

c. Save The Children Foundation 

d. Don't know/Not sure 

Characters Who Are Supposed to be Typical of the Target Audience: 

(The word "characters" in these questions could be substituted with "man." "woman," "family," "children," 
etc.) 

1. Which of the following statements better describes the characters in the announcement? 

a. The characters in the announcement reminded me of people I know. 

b. The characters in the announcement did not remind me of people I know. 

2 Overall, how would you describe the characters in the announcement? Please select one response from 
each group. 

a. Realistic 

b. Not realistic 

a. Helped me to understand the message 

b. Did not help me to understand 

3 Overall, how would you describe *he characters in the announcement? Please select one answer from each 
group. 

a. Appealing 

b. Not appealing 

a. Get the message across 

b. Do not get the message across 

a. Believable 

b. Not believable 

a. Easy to understand 

b. Not easy to understand 

Use of a VDice-over Announcer 

1. Please circle the one answer from each pair of phrases which better describes your feelings about the 
announcer. 

a. Believable 

b. Not believable 

a. Appropriate to the message 

b. Not appropriate to the message 

a. Gets the message across 

b. Does not get the messago across 

a. Easy to understand 

b. Hard to understand 
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Presentation of Controversial or Unpleasant Infonnation: 

1. Some people have mentioned different feelings they had during or after watching the announcement Please 
circle the opinion which comes closest to yours. 

a. The announcement made me uncomfortable and I had difficulty paying attention to it. 

b. The announcement interested me so I paid attention to it. 

c. I had no particular feeling about the announcement. 

2. Overall, how do you think most people would feel about this announcement if they saw it on television at 
home? 

a. Suitable to show on television at any time 

b. Suitable to show, but only at certain tinnes 

c. Not suitable to show at any time 

Listed below are sample questions to be used for identifying the characteristics of your pretest participants 
Some of these questions also may be used as screening questions to recruit participants from a specific target 
group. 

1 What is your sex? 

a. Male 

b. Female 

2. What is your age? 



a. 


Under 18 


b. 


18-24 


c. 


25-34 


d. 


35-44 


e. 


45-49 


f. 


50-54 


g- 


55-60 


h. 


Over 60 



3. How far did you go in school? 

a. Eighth grade or less 

b. Some high school 

c. High school graduate 

d. Some college 

e. College graduate 
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4. Do you have children? 

a. Yes (go to question 4a) 

b. No (go to question 5) 

4a.Please circle the age categories in which your children belong. Circle as many as apply. 

a. 1-5 years old 

b. 6-10 years old 

c. 1M5 years old 

d. 16-20 years old 

e. 21 and over 

5. Which of the following statements best describes you? 

a. I currently smoke. 

b. I used to smoke, but have now stopped. 

c. I have never smoked. 

6. Have you ever been told by a doctor or a nurse that you have . . . 

a. Heart disease 

b. High blood pressure 

c. Cancer 

d. Emphyzema 

e. Other 

7. To the best of yo'jr knowledge, have you ever been exposed to . . , 

a. Asbestos 

b. Other toxic chemicals 

c. Etc. 

8. Which of the following best describes your race or ethnic background? 
a. White 

b Black 
c. Hispanic 
d Asian 
e. Other 
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Script for Hosting the Theater Test 



Introduction 

Good evening, I'm and I'd like to thank you all for coming today (tonight). 

I think we're all going to have a good time. Just a few more words of introduction and we'll get started. We've 
asked you here because we feel that it's very important to get your i about new television program 
material. So consider this your chance to give the TV program pror* your opinions . . . tiefore the program 
gets to your home screens. 

Keep in mind that what we're interested in is your own personal views. We don't want you to tell us what you 
think we want to hear or what your spouse thinks or anyone else — we need your own opinions. 

There are no right answers and no wrong answers. So, please don't discuss the program or your answers 
With the people around you. 

Also, please make sure that you write your answers clearly in the space provided on the questionnaire we've 
given you, and be sure that you don't move to each new section of the questionnaire until I tell you to 

Do you have any questior»s about this procedure? I'll answer any other questions you have at the end of the 
test session. Okay? Now enjoy the show. 

(Play the pretesting program videotape) 



Reactions to the Program 

All right. Now we'd like to find out about your reactions to the program— what parts you liked and what parts 
you didn't like. Now, I'd like you to turn to the first page of the questionnaire and answer the questions. Please 
don't go on to the next page until I tell you to. 
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Attention. A pretesting measure to 
describe a message's ability to attract 
listener or viewer attention, this is 
often called "recall." 

Attitudes. An individuars predis- 
positions toward an object, person or 
group, which Influence ms or her 
response to be either positive or 
negative, favorable or unfavorable, etc. 

Baseline study. The collection and 
analysis of data regarding a target 
audience or situation prior to inter- 
vention. 

Central location intercept 
interviews. Interviews conducted with 
respondents who are stopped at a 
highly trafficked location that is 
frequented by individuals typical of the 
desired target audience. 

Channel. The route of message 
delivery (e.g., mass media, community 
interpersonal) 

Closed-ended questions. Questions 
that provide respondents with a list of 
possible answers from which to 
choose, ^' :o called multiple choice 
questicis. 

Communication concepts. Rough art 
work and statements that convey the 
Idea for a full message. 

Communication strategy 
statement. A written statement that 
includes program objectives, trrget 
audiences, an understanding of the 
information needs and perceptions of 
each target audience; what actions 
they should take, the reasons why they 
should act and the benefits to be 
gained. This document provides the 
direction and consistency for all 
program messages and matenals. 

Comparison group. A control group 
randomly selected and matched to the 
target population according to 
characteristics identified in the study 
to permit a comparison of changes 
between those who receive the 
intervention and those who do not. 

Comprehension. A pretesting 
measure to determine whether 
messages are clearly understood. 



Convenience samples. Samples that 
consist of respondents who are typical 
of the target audience and who are 
easily accessible; not statistically 
projectable to the entire population 
being studied. 

Diagnostic informrii!on. Results from 
pretesting research that indicate the 
strengths and weaknesses in 
messages and materials. 

Focus group interviews. A type of 
qualitative research in which an 
experienced moderator leads about 8 
to 10 respondents through a 
discussion of a selected topic, allowing 
them to talk freely and spontaneously. 

Formative evaluation. Evaluative 
research conducted during program 
development. May include state-of-the- 
art reviews, pretesting messages and 
materials, and pilot testing a program 
on a small scale before full imple- 
mentation. 

Frequency. In advertising, is used to 
describe the average number of times 
an audience is exposed to a specific 
media message. 

Gatelceeper. Someone you must work 
with before you can reach a target 
audience (e.g., a schoolteacher) or 
accomplish a task (e.g., a television 
public service director) 

Goal. The overall improvement the 
program will otrive to create. 

Impact Evaluation. Research de- 
signed to identify whether and to what 
e^^ent a program contnbuted to 
accomplishing its stated goals (here, 
more global than outcome evalua- 
tion). 

In-depth interviews. A form of 
qualitative research consisting of 
intensive interviews to find out how 
people think and what they feel about 
a given topic. 



Intermediaries. Organizatioris, such 
as professional, Industrial, civic, social 
or fraternal groups, that act as 
channels for distributing program 
messages and materials to members 
of the desired target audience. 

Objectiva A quantifiable statement of 
a desired program achievement 
necessary to reach a program goal. 

Open-ended questions. Questions 
that allow an individual to respond 
freely in his or her own words. 

Outcome evaluation. Research 
designed to account for a program's 
accomplishments and effectiveness; 
also called '"impact" evaluation. 

Over-recruiting. Recruit.ng more 
respondents than required to 
compensate for expected '*no-shows." 

Polysyllabic words. Words that 
conta n three or more syllables. 

Pretesting. A type of formative 
research that involves systematically 
gathering target audience reactions to 
messages and materials before they 
are produced in final form. 

Probe. Interviewer technique used to 
solicit additional information about a 
question or issue. Probe should be 
neutral (e.g.. "What else can you tell 

me about ?'*) not directive ("Do 

you think the pamphlet was suggesting 
that you take a particular step — such 
as changing your diet?") 

Process evaluation. Evaluation to 
study the functioning of components of 
program implementation; includes 
assessments of whether materials are 
being distributed to the right people 
and in what quantities, whether and to 
what extent program activities are 
occurring, and other measures of how 
and how well the program is working 

PSA. Public service announcement, 
used without charge by the media. 
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Qualitative research. Research that 
IS subjective in that it involves ob- 
taining infornnation about feelings and 
impressions f ronn snnall nunnbers of 
respondents. The information gathered 
usually should not be described in 
numerical terms, and generalizations 
about the target populations should not 
be made 

Quantitative research. Research de- 
signed to gat her objective information 
from representative, random samples 
of respondents; results are expressed 
in nunnerical terms (e.g., 35 oercent 
are aware of X and 65 percent are 
not) Quantitative data are used to 
draw conclusions about the target 
audience. 

Random sample. A sample of 
respondents in which every individual 
of the population has had an equal 
chance of being included in the 
sample. 



Reach. In advertising, used to 
describe the number of different 
people or households exposed to a 
specific media message during a 
specific period of time. 

Readability testing. Using a formula 
to predict the approximate reading 
grade level a person must have 
achieved in order to understand 
written materia! 

Recall. In pretesting, used to descnbe 
the extent to which respondents 
remember seeing or hearing a 
message that was shown in a conn- 
petitive media environment, usually 
centers on nnain idea or copy point 
recall. 

Segmentation. Subdividing an overall 
population .nto homogeneous subsets 
of target a. diences m order to better 
describe and understand a segment, 
predict behavior, and formulate tailored 
messages and programs to meet 
specific needs. Segments may be 
demographic (eg., age, sex. education, 
family life cycle), geographic (e.g.. 
Southeastern, U.S., rural, north side of 
town) or psychographic (e.g., 
personality lifestyle, usage patterns, 
risk factors, benefits sought). 

Self-administered questionnaires. 

Questionnaires that are filled out by 
respondents themselves; mailed 
directly to the respondent, or filled out 
by respondents gathered at a central 
location 



Social marketing. A discipline that 
addresses an issue with particular 
regard to those affected by it (the 
target audience), considering their 
perspectives and perceived wants and 
needs to develop strategies toward 
change. 

Summative evaluation. Evaluation 
conducted to identify a program's 
accomplishments and effectiveness, 
also called outcome or impact 
evaluation. 

Target audience The desired or 
intended audience for program mes- 
sages and materials (see segmenta- 
tion). The primary target a'jdience 
consists of those individuals the 
program is designed to affect. The 
secondary target audience is that 
group (or groups) that can help reach 
or influence the primary audience 
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e would appreciate any comments or suggestions 
you have about this guide; such suggestions will 
be used in preparing future editions of the publi- 



Thank you. 



Please send to: 



cation. 



John Burklow 

Office of Cancer Communications, NCI 
Building 31, Room4B43 
Bethesda, MD 20892 



1. How much of this book did you read'? 
All of it Some of it C 



Did not read it 



2. If you read some of the workbook, please mark which 
chapters you read: 

Introduction 

The Health Communication Process 

Stage 1: Planning and Strategy Selection 

Stage 2: Selecting Channels and Pretesting 

Stage 3: Developing Materials and Pretesting 

Stage 4: Implementing Your Program 

Stage 5: Assessing Effectiveness 

Stage 6: Feedback to Refine Program 

3. Did yDu find the workbook to be 

very useful somewhat useful not useful 

4. Please circle those chapters listed above that you 
found most useful. 

5. How have you used this book? (check as many as 
apply) 

Personal Reference 

Student Instruction 

Staff Development 

Library Resource 

Other 

6. Your job title? 

7. Affiliation or employer? , 

8 How might this handbook be improved? 
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